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Preface 


The  purpose  of  this  study  was  to  analyze  the  Engineering  Change 
Proposal  (ECP)  process  in  the  Advanced  Cruise  Missile  (ACM)  System 
Program  Office  (SPO) .  The  ACM  is  critical  to  maintaining  a  level  of 
strategic  deterrence  for  the  United  States,  and  the  ECP  process  was 
perceived  to  be  untimely.  For  these  two  reasons  I  chose  to  analyze  the 
ECP  process  in  the  ACM  SPO  to  develop  recommendations  for  improving  its 
timeliness.  I  statistically  analyzed  actual  SPO  process  time  data  and 
the  results  of  a  survey  questionnaire  administered  to  ACM  SPO  personnel, 
and  I  concluded  that  the  untimeliness  of  certain  process  steps  was  a 
symptom  of  a  lack  of  effort  or  concern  on  the  part  of  the  ACM  SPO  per¬ 
sonnel  during  the  review  of  the  Technical  Change  Package  (TCP),  a  pre¬ 
cursor  document  to  the  ECP.  The  time  needed  to  perform  rework  was  found 
to  be  significant,  and  I  recommended  that  the  SPO  institute  programs  to 
increase  the  awareness  of  the  need  for  improvement  in  this  area. 

This  thesis  could  not  have  been  completed  without  the  help  from 
others.  First,  Professor  Daniel  Reynolds  pro/ided  untiring  patience, 
tremendous  assistance,  and  wonderful  guidance  throughout  the  thesis 
process.  Personnel  in  the  ACM  SPO  also  deserve  credit  for  their 
assistance.  Robert  Connolly  and  Marilyn  Judd  provided  outstanding 
technical  assistance,  Doug  Jones  invited  me  to  participate  on  th-. 
Critical  Process  Improvement  Team,  and  Col  Bolton  and  Lt  Col  Hemmig 
sponsored  my  effort  in  the  SPO.  Last,  but  certainly  not  least,  my  wife 
Roxanne  was  a  tremendous  help  to  me  through  the  long  days  and  nights 
that  I  spent  preparing  this  report. 

Jeffery  S.  Robertson 
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Abstract 

This  study  investigated  the  engineering  change  proposal  (ECP) 
process  in  the  Advanced  Cruise  Missile  (ACM)  System  Program  Office 
(SPO).  Specifically,  efforts  were  geared  toward  verifying  the  process 
untimeliness  and  identifying  the  specific  causes  of  the  untimeliness. 
Also,  a  survey  questionnaire  was  analyzed  to  assist  in  the  development 
of  conclusions  about  the  problems  with  the  ECP  process.  Specific 
process  steps  were  identified  as  untimely;  however,  the  major  conclusion 
was  that  the  personnel  in  the  ACM  SPO  were  not  taking  advantage  of  the 
opportunity  to  review  the  Technical  Change  Package  (TCP) ,  a  preliminary 
copy  of  the  ECP.  By  doing  so,  the  SPO  perhaps  could  have  produced 
better  quality  ECPs  which  the  approval  authorities  could  have  approved 
as  written.  Instead,  because  of  content  and  documentation  errors,  the 
ECPs  had  to  be  deferred  or  approved  with  comments,  forcing  the  SPO  to 
perform  additional  process  steps  which  added  significantly  to  overall 
processing  time  required  for  eventual  approval.  The  recommendation  was 
made  for  the  SPO  to  educate  the  personnel  about  the  importance  of  an 
adequate  TCP  review,  encourage  imp  ovement  through  the  adoption  of  an 
awards  program,  and  take  steps  to  educate  those  personnel  in  the  SPO  who 
need  training. 


IX 


AN  ANALYSIS  OF  THE  ADVANCED  CRUISE  MISSILE  SYSTEM  PROGRAM 


OFFICE  ENGINEERING  CHANGE  PROPOSAL  PROCESS 

I ,  Introduction 

General  Background  Information 

Global  Scenario.  Since  the  end  of  World  War  II  the  Soviet  Union 
has  been  the  only  nation  with  the  capability  to  threaten  the  national 
survival  of  the  United  States  (8:1).  Because  of  this,  the  United  States 
has  focused  its  national  strategy  to  counter  this  capability  (8:1). 
Furthermore,  the  United  States  and  the  Soviet  Union  support  different 
ideologies,  with  the  Soviet  Union  supporting  the  spread  of  communism  and 
the  United  States  trying  to  contain  the  spread  (26:1-5).  Although  the 
two  world  powers  have  fought  against  each  other’s  ideologies  indirectly, 
in  Korea  and  Vietnam  for  example,  they  have  not  been  involved  in  a 
direct  exchange  of  fire  on  a  major  scale.  Why  not?  After  all,  both 
countries  possess  the  capability  to  destroy  each  other's  industrial  base 
(25:1).  Perhaps  the  two  have  not  exchanged  direct  fire  because  of  the 
development  and  implementation  of  effective  national  security  strate¬ 
gies  (NSS)  by  the  United  States  presidential  administrations  since  World 
War  II.  From  President  Truman's  policy  of  "Containment"  to  Eisenhower's 
"Massive  Retaliation"  and  Kennedy's  and  Johnson's  "Mutually  Assured 
Destruction"  (MAD)  policies,  the  l  'ited  States  relied  heavily  on  its 
nuclear  arms  superiority  over  the  Soviet  Union  to  deter  hostile  actions 
from  it  (26:1-3;.  However,  because  of  an  escalating  growth  in  the  nuc- 
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lear  arms  capabilities  of  the  Soviet  Union,  the  presidential  administra¬ 
tions  from  Nixon  and  on  have  relied  on  a  "Realistic  Deterrence"  NSS 
focusing  on  detente  and  preplanned  and  proportional  nuclear  responses  to 
aggressive  behavior  (26:3-5).  (Note:  although  President  Reagan's  NSS 
was  named  "Flexible  Response,”  it  retained  the  essential  characteristics 
of  the  former  Realistic  Deterrence  strategy  (26:5).)  These  strategies 
have  remained  fairly  constant  through  the  years,  despite  the  changes  in 
the  names,  and  they  place  heavy  emphasis  on  deterrence  to  war  through 
the  availability  of  effective  strategic  nuclear  forces  (26:4-5). 

The  United  States'  Efforts.  With  the  evolution  of  the  United 
States  national  security  strategy,  to  include  not  only  "assured  destruc¬ 
tion"  but  also  deterrence  of  limited  nuclear  attacks,  the  development  of 
effective  strategic  offensive  forces  has  been  crucial  (26:1).  Indeed, 
"maintaining  stable  nuclear  deterrence  vis  a  vis  the  Soviet  Union  will 
depend  on  ...  a  modernized  mix  of  nuclear  systems  that  hold  critical 
Soviet  assets  at  risk"  (8:3).  This  focus  has  resulted  in  the  develop¬ 
ment  of  strategic  nuclear  ballistic  missiles,  air-  and  sea-launched 
nuclear  cruise  missiles,  and  the  delivery  and  command  and  control 
systems  which  get  them  to  where  they  need  to  go  (25:1). 

Furthermore,  the  United  States  has  developed  a  strategic  deter¬ 
rence  strategy  to  magnify  its  overall  deterrent  capability  (25:2).  This 
strategy  consists  of  a  focus  on  a  TRIAD  of  weapons  and  delivery  methods, 
including  land- launched  intercontinental  ballistic  missiles  (ICBMs) , 
submar ine - launched  ballistic  missiles,  and  manned  bombers  which  use  nuc¬ 
lear  bombs  and  air-launched  cruise  missiles  (25:2).  The  three  different 
legs  of  the  TRIAD  offer  different  advantages,  plus  the  overall  advantage 
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of  forcing  the  Soviet  Union  to  spend  resources  to  defend  against  all 
three  (25:2).  Land-based  missiles  are  very  accurate,  have  a  high  alert 
rate,  plus  other  advantages,  but  they  cannot  be  overtly  displayed  as  a 
show  of  force--once  they  are  launched  they  cannot  be  called  back  (25:2). 
Likewise,  submarine -launched  ballistic  missiles  are  accurate,  and  they 
are  the  most  survivable  element  of  the  TRIAD  [unless  the  submarine  can 
be  found  before  its  missiles  are  launched]  (25:4-5).  Submarines,  how¬ 
ever,  are  not  very  suitable  for  use  as  a  show  of  force  and  will,  since 
their  mission  is  to  remain  in  hiding  (25:5).  These  two  missile  systems 
leave  a  gap  in  capability  that  is  filled  by  manned  strategic  bombers 
(25:5).  Indeed, 

Manned  aircraft  provide  the  highest  measure  of  controllability  and 
flexibility- -  the  capability  to  respond  to  a  wide  variety  of  un¬ 
foreseen  and  rapidly  changing  circumstances.  They  can  be  launched 
for  survival  or  a  show  of  force  or  intent  and  can  be  recalled  if 
execution  is  not  directed.  Manned  aircraft  can  hunt  out  and 
destroy  targets  that  cannot  be  located  precisely  in  advance. . . . 
They  can  react  immediately  to  redirection,  exploit  fleeting 
advantages,  and  execute  a  broad  range  of  missions.  They  offer  the 
vital  power  of  human  observation  and  evaluation.  (25:2) 

Moreover,  using  guided  cruise  missiles  (ALCMs) ,  manned  bombers  have  the 

"capability  to  deliver  massive  firepower  at  long  ranges"  (25:6). 

The  [ALCM]  is  a  small,  unmanned,  winged  air  vehicle  capable  of 
sustained  subsonic  flight  following  launch  from  a  carrier  air¬ 
craft.  It  has  a  turbofan  engine  and  a  nuclear  warhead  and  is 
programmed  for  precision  attack  on  surface  targets.  When  launched 
in  large  numbers,  each  of  the  missiles  would  have  to  be  countered, 
making  defense  against  them  both  costly  and  complicated.  Addi¬ 
tionally,  by  diluting  defenses,  the  ability  of  manned  aircraft  to 
penetrate  to  major  targets  is  improved.  Small  radar  signature  and 
low-level  flight  capability  enhance  the  missile’s  effectiveness . 
(31:156) 

However,  cruise  missiles  and  manned  aircraft  are  not  immune  to  advance¬ 
ments  in  defenses,  especially  the  recent  advancements  made  by  the  Soviet 
Union . 
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The  Soviet  Union's  Efforts.  Meanwhile,  the  Soviet  Union  has 


developed  not  only  their  own  TRIAD,  but  also  countermeasures  to  United 
States  capabilities.  These  countermeasures  include  anti-aircraft  and 
anti-cruise  missile  defensive  systems  around  their  political  and  indus¬ 
trial  centers.  The  Los  Angeles  Times  reported  on  August  7,  1983  that 
"Soviet  air  defenses,  already  the  world's  most  advanced,  are  steadily 
improving,  according  to  ’Soviet  Military  Power,'  a  Defense  Department 
document  released  last  March"  (29:28).  Furthermore, 

[Soviet]  airborne  radar  can  find  a  present-generation  cruise 
missile  no  matter  how  low  it  flies  or  how  much  it  bobs  and  weaves. 
The  Soviets'  Foxhound  A  fighter  aircraft  is  said  to  have  this 
"look-down/shoot-down"  capability  and  Soviet  versions  of  the  huge 
American  AWACS  radar  surveillance  planes,  with  even  better  detec¬ 
tion  capability,  are  said  to  be  improving.  Another  Soviet  threat 
is  the  SA-10,  a  surface-to-air  missile  that  might  be  able  to  knock 
down  a  slow-flying  cruise  missile  "at  any  altitude,"  according  to 
"Soviet  Military  Power."  (29:28) 

To  counter  these  advancements  by  the  Soviet  Union,  the  United  States  has 
chosen  to  produce  and  field  advancements  of  its  own  so  that  it  may 
achieve  one  of  its  goals  (8:3).  What  goal  is  important  enough  to  the 
United  States  that  it  would  be  willing  to  spend  large  amounts  of 
resources  to  achieve? 

An  Overall  Goal .  A  stated  goal  of  the  United  States  is  to  main¬ 
tain  peace  with  the  Soviet  Union  (26:1-5).  A  means  by  which  this  goal 
can  be  achieved  is  to  continue  to  develop  and  deploy  weapons  which 
support  the  three  legs  of  the  TRIAD  so  that  an  effective  deterrent 
capability  continues  to  exit  (25:1).  The  overall  goal  as  stated  here 
assumes  that  war  between  the  United  States  and  the  Soviet  Union  is  and 
will  always  be  less  desirable  than  peace.  Also,  as  long  as  the  Soviet 
Union  is  swayed  toward  maintaining  peace  because  of  our  nuclear  weapons, 
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then  a  means  to  maintain  that  peace  is  through  the  continued  development 
and  deployment  of  these  types  of  weapons. 

The  Overall  Requirement.  So,  based  on  the  above  assumptions,  the 
United  States  is  required  to  have  effective  deterrent  forces  including 
land-,  sea-,  and  air-based  nuclear  weapons  which  are,  in  the  eyes  of  the 
Soviet  Union,  powerful,  accurate,  and  able  to  reach  their  targets  (8:3). 
Recall  from  the  discussion  above  that  the  United  States 's  land- and  sea- 
based  systems  are  relatively  more  accurate  and  more  survivable  than  air- 
based  systems,  but  cannot  be  used  efficiently  as  a  show  of  force  and 
will  (25:2,5).  Also,  because  of  continuing  improvements  in  Soviet 
Union's  anti-aircraft  and  anti-cruise  missile  defensive  systems,  the 
United  States  has  decided  to  meet  the  overall  need  by  focusing  its 
attention  on  improving  the  air-based  leg  of  the  TRIAD  in  an  effort  to 
deploy  a  newer  weapon  system  which  will  be  more  survivable  than  the 
older  (31:156).  This  older  weapon  is  the  Air-Launched  Cruise  Missile 
(ALCM) ,  which  was  first  fielded  in  1980  (3:4).  The  newer  weapon  is  the 
Advanced  Cruise  Missile  (ACM)  (3:3).  The  United  States  initiated  a 
development  program  for  the  ACM  in  1983,  with  the  intent  that  a  more 
survivable,  advanced  [technology]  cruise  missile,  would  replace  the 
older  generation  ALCM  (31:156).  Hence,  to  meet  the  need,  a  major 
requirement  was  born  to  develop  and  deploy  an  Advanced  Cruise  Missile 
that  will  survive  the  Soviet  defensive  systems,  reach  the  target,  and 
detonate.  The  flow  from  overall  goal  to  the  major  requirement  to  deploy 
the  ACM  is  shown  in  Figure  1.  Furthermore,  Figure  2  continues  the  train 
of  thought  from  the  establishment  of  the  ACM  SPO  goal  to  a  specific  ACM 
SPO  requirement. 
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Figure  1.  Flow  from  Overall  Goal  to  Major  Requirement 


ACM  SPO  Goal  and  Requirement 

To  support  this  major  requirement,  the  Advanced  Cruise  Missile 
(ACM)  System  Program  Office  (SPO)  has  adopted  goals  that  stress  the  need 
to  "provide  the  User  with  a  system  that  meets  all  stated  requirements" 
[including  timely  hardware  deliveries]  in  a  cost-effective  manner 
(30:3).  Indeed,  the  ACM  SPO  has  been  tasked  with  the  responsibility  of 
performing  the  necessary  weapons  system  acquisition  steps  to  make  this 
requirement  a  reality  (10:4-8).  These  acquisition  steps  stress  the 
importance  of  basing  decisions  on  the  tradeoffs  between  the  aspects  of 
cost,  schedule,  performance,  and  logistics  supportability  (19:10). 

Since  these  aspects  must  be  balanced,  it  follows  that  the  ACM  SPO  needs 
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Timely  ECP  Process 


Figure  2.  Flow  from  Major  SPO  Goal  to  Specific  Requirement 


to  have  timely  and  efficient  internal  processes  which  do  not  impact 
these  aspects.  Indeed,  the  ACM  SPO  management  would  perhaps  be 
concerned  about  the  efficiency  and  timeliness  of  a  process  in  the  SPO 
that  could  negatively  impact  each  of  these  four  aspects  of  the  program. 

Specific  Requirement.  One  such  process  is  the  Engineering  Change 
Proposal  (ECP)  process.  ECPs  are  generated  by  the  contractor,  approved 
by  the  government,  and  executed  by  the  contractor;  they  are  documenta¬ 
tion  packages  which  include  the  necessary  details  for  making  changes  to 
the  contractual  technical  baseline  (7:2).  The  steps  involved  in  ECP 
generation,  review,  approval,  and  implementation  comprise  the  ECP 
process  (6:3)  .  If  the  ECP  process  is  not  functioning  in  a  timely 
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manner,  then  untimely  ECP  processing  may  result  in  higher  costs  to  the 
government.  These  costs  include  not  only  personnel  resources  lost  due 
to  inefficient  and  untimely  ECP  processing,  but  also  higher  prices  for 
implementing  changes  in  hardware  later  as  opposed  to  sooner.  Also, 
proposed  solutions  to  performance  or  logistics  supportability  problems 
or  enhancements  that  require  the  technical  baseline  to  be  changed  might 
have  to  be  implemented  later  than  desired  because  of  untimely  ECP 
processing.  This  delay  could  negatively  impact  the  scheduled  delivery 
of  performance  or  supportability  improvements  that  are  needed  to  meet  a 
certain  level  of  capability.  Thus  a  specific  requirement  exists  to  have 
an  ECP  process  that  is  both  efficient  and  timely. 

The  Problem  Statement  and  Motivation  of  the  Study 

A  Perceived  Problem.  However,  based  mainly  on  the  results  of 
informal  and  undocumented  internal  time  studies,  the  ACM  SPO  director, 
project  managers,  and  functional  managers  have  questioned  the  timeliness 
of  the  ECP  process  (16).  Furthermore,  the  process  has  recently  been 
identified  by  the  ACM  managers  as  an  "ACM  critical  process,"  one  that  is 
critical  to  program  success  (16).  The  managers  have  formed  a  critical 
process  improvement  team  (CPIT)  to  study  the  overall  ECP  process  in  the 
ACM  SPO  and  to  make  recommendations  regarding  the  process'  efficiency 
and  timeliness  (16) . 

Problem  Statement.  Based  on  the  perception  that  the  process  is 
potentially  untimely,  the  specific  problem  this  thesis  will  study  is  the 
timeliness,  or  lack  thereof,  of  the  ACM  SPO  ECP  process. 

Motivation  of  the  Hypotheses.  Indeed,  if  the  ECP  process  is 
untimely,  then  the  process  needs  to  be  improved  so  that  the  United 
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States  does  not  have  to  wait  any  longer  than  necessary  to  field  this 
deterrent  weapon  and  so  that  the  additional  costs  associated  with 
untimely  and  inefficient  processes  will  be  saved.  Furthermore,  specific 
requirements  must  be  met  if  a  process  like  the  ACM  SPO  ECP  process  is  to 
be  timely.  For  example,  a  process  involving  many  people  requires  not 
only  sufficient  numbers  of  people  available  to  handle  their  workloads, 
but  also  people  who  are  willing  and  able  to  communicate  with  one 
another.  Furthermore,  since  an  ECP  is  a  document,  the  preparer  of  the 
ECP  must  be  able  to  effectively  describe  the  need  in  words,  and  the 
government  personnel  must  make  every  effort  to  assist  in  the  preparation 
process.  These  specific  requirements  may  be  lacking  in  the  ACM  SPO,  and 
they  lead  to  the  set  of  hypotheses  that  the  researcher  has  developed  and 
discusses  below. 

Tentative  Hypotheses 

In  order  to  determine  whether  or  not  the  ECP  process  in  the  ACM 
SPO  is  untimely,  hypotheses  will  be  made  and  tested  using  the  methods 
outlined  in  Chapter  III.  The  researcher  has  constructed  two  areas  of 
hypotheses,  and  the  presentation  of  these  hypotheses  will  follow  the 
general  flow  pictured  in  Figure  3.  First,  the  perceived  timeliness 
problem  will  be  studied  by  analyzing  a  hypothesis  about  the  timeliness 
of  the  ECP  process.  The  second  set  of  hypotheses  will  be  based  on  the 
requirements  for  timely  processing,  stated  above,  and  on  the  assumption 
that  the  data  indicates  a  timeliness  problem. 

The  First  Hypothesis.  The  first  hypothesis  states  simply  that  the 
ACM  SPO  ECP  process  is  not  timely.  This  hypothesis  will  be  tested  by 
comparing  actual  times  to  complete  process  steps  to  the  amount  of  time 


9 


ADEQUATE  UP¬ 
FRONT  EFFORT 


Figure  3.  Flow  of  Presentation  of  Hypotheses 


allowed  in  the  ACM  SPO  Configuration  Management  Plan  (CMP)  which  is  a 
contractual  document  between  the  ACM  SPO  and  the  contractor. 

The  Second  Set  of  Hypotheses.  For  this  second  set  of  hypotheses, 
we  make  the  assumption  that  the  ECP  process  has  already  been  found  to  be 
untimely.  Then,  if  the  ACM  SPO  is  to  improve  the  timeliness  with  which 
it  implements  ECPs ,  it  must  identify  the  requirements  for  effective  and 
efficient  ECP  processing.  Four  hypotheses  are  proposed  which  are 
potential  barriers  to  timely  ECP  processing. 

1.  Communication.  The  ACM  SPO  consists  of  about  180 
personnel  in  13  different  offices,  or  directorates.  Approximately  80  of 
these  personnel  are  involved  in  the  ECP  process  at  any  one  time,  each 
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involved  with  the  processing  of  any  number  of  ECPs  from  one  to  30. 
Coordinating  the  efforts  of  the  individuals  involved  in  the  process  is  a 
complex  task,  and  it  is  one  which  justifies  this  first  hypothesis: 

ACM  SPO  personnel  do  not  sufficiently  communicate  with  other  ACM  program 
personnel  within  or  outside  the  SPO.  Potential  causes  of  the  lack  of 
communication  are  numerous- -two  examples  follow:  excessive  numbers  of 
temporary  duty  (TDY)  assignments  prevent  ACM  SPO  personnel  from  being 
available  to  discuss  issues,  or  workers  simply  fail  to  discuss  their 
efforts  with  th  ir  supervisors.  This  lack  of  communication  between 
subordinates  and  supervisors  results  in  uncoordinated,  delayed,  or 
wasted  efforts,  especially  when  subordinates  do  not  communicate  with 
members  of  the  SPO  who  are  involved  in  the  E CP  approval  process. 

2.  ECP  Submissions.  A  second  requirement  exists  if  ECP 
processing  is  to  be  timely.  Since  the  contractor  is  required  to  draft, 
edit,  and  submit  ECPs  to  the  ACM  SPO,  if  the  submittal  is  poorly  written 
or  conveys  a  message  which  is  contrary  to  what  is  intended,  then  either 
time  will  be  spent  interpreting  and  rewriting  the  submittal  or  the 
discrepancy  will  go  unnoticed.  This  requirement  for  clearly  written 
ECPs  is  perhaps  not  being  met,  which  leads  to  the  second  hypothesis: 

The  contractor's  ECP  submissions  are  unclear,  lack  sufficient  detail, 
and  are  inaccurate .  Conflicts  between  what  is  stated  in  the  ECP  and 
what  is  actually  desired  or  required  increase  the  time  needed  to  iron 
out  the  differences  and  delay  ECP  approval. 

3,  Personnel  Resources.  A  third  requirement  of  the  ACM  SPO 
is  people.  People  are  needed  to  process  the  paperwork  associated  with 
ECPs  and  to  perform  other  functions  as  well.  However,  if  the  number  of 
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personnel  or  the  level  of  training  of  those  personnel  are  insufficient 
to  adequately  perform  their  assignments,  then  a  third  problem,  inherent 
in  the  third  hypothesis,  will  exist:  The  ACM  SPO  lacks  adequate  person¬ 
nel  resources  to  handle  the  number  of  ECPs  in  the  ECP  process  at  any  one 
time.  Without  adequate  personnel  resources,  either  the  length  of  time 
required  for  review  and  implementation  of  the  ECPs  by  the  appropriate 
personnel  increases  or  the  time  remains  the  same  but  the  effectiveness 
of  the  operation  diminishes . 

4.  Preparatory  Work  at  TCMs .  Finally,  although  the  number 
of  personnel  and  the  level  of  training  may  be  sufficient,  if  the  person¬ 
nel  do  not  adequately  review  the  Technical  Change  Package  (TCP) ,  a  docu¬ 
ment  from  which  the  ECP  is  prepared,  before  the  Technical  Coordination 
Meeting  (TCM) ,  the  meeting  at  which  the  TCP  is  reviewed  with  the 
contractor,  then  the  ECP  may  have  content  and  or  documentation  errors 
which  could  delay  approval  of  the  ECP.  The  fourth  hypothesis  is  a 
statement  about  the  TCP  and  TCM  review  effort:  Insufficient  up-front 
effort  by  government  personnel  at  Technical  Coordination  Meetings 
results  in  time  delays.  Again,  if  the  TCP  is  not  reviewed  by  all  of  the 
functional  specialists,  or  if  the  person  reviewing  the  TCP  does  not 
attend  the  TCM,  then  the  ECP  could  be  submitted  to  the  government  with 
content  or  documentation  errors  which  will  need  to  be  fixed  before  the 
ECP  can  be  approved.  This  "fixing"  process  could  take  extra  time,  time 
which  could  result  in  delayed  implementation  of  hardware  enhancements. 

Justification 

Why  is  the  ACM  SPO  ECP  process  being  studied?  The  answer  lies  in 
the  importance  of  the  ECP  process  to  the  United  States.  Recall,  a  major 
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overall  goal  of  the  United  States  is  to  maintain  peace  with  the  Soviet 
Union  (26:1-5).  To  keep  the  peace,  the  United  States  is  relying  on  a 
strong  deterrent  force  consisting  of  ground-,  sea-,  and  air-based 
weapons  and  delivery  systems  (26:4-5).  However,  the  effectiveness  of 
the  air-based  portion  of  the  TRIAD  has  grown  weaker  as  the  Soviet  Union 
has  made  advancements  in  its  defensive  capabilities  (29:28).  The  United 
States  has  decided  to  improve  the  deterrent  capability  of  this  portion 
of  the  TRIAD  through  development  and  deployment  of  the  ACM  (31:156). 

The  ACM  SPO  has  been  tasked  to  develop  and  deliver  to  SAC  the  ACM  as 
rapidly  as  possible  (10:1-4).  To  meet  its  goal  of  timely  deliveries  to 
SAC,  the  ACM  SPO  is  required  to  have  timely  and  efficient  processes. 
However,  the  ECP  process  has  been  perceived  to  be  untimely  (16).  Also, 
the  ECP  process  is  believed  to  be  delaying  the  delivery  of  a  level  of 
deterrent  capability  to  the  user,  decreasing  ACM  SPO  productivity,  and 
increasing  costs  (16).  Furthermore,  the  defense  industry  can  ill  afford 
inefficient  and  costly  operations  in  this  age  of  defense  budget  cuts  and 
manpower  reductions.  Through  implementation  of  research  findings,  it  is 
anticipated  that  the  magnitude  of  all  of  these  problems  can  be  reduced. 

Scope  of  Research  and  Limitations  of  Findings 

Scope  of  Research.  Although  there  are  other  internal  SPO  proc¬ 
esses  in  the  ACM  SPO,  this  research  effort  will  be  limited  to  a  study  of 
the  ECP  process.  Furthermore,  although  other  problems  may  exist  within 
the  ACM  SPO  ECP  process,  this  thesis  will  be  limited  to  analyzing  only 
the  five  hypotheses  listed  above.  Moreover,  a  specific  time  period  will 
be  studied,  including  the  period  from  June  1988  through  August  1990;  it 
is  possible  that  different  conclusions  could  be  obtained  by  studying  a 
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different  period  of  time.  Finally,  even  though  many  different  organiza¬ 
tions  are  involved  in  the  ACM  SPO  ECP  process,  this  study  will  concen¬ 
trate  primarily  on  the  interactions  of  the  ACM  SPO  personnel  in  and 
among  themselves.  All  of  these  limitations  serve  to  narrow  the  scope  of 
the  problem  so  the  specific  problem  can  be  studied  in  more  detail. 

Limitations  of  Findings.  The  proposed  solutions  to  improving  the 
timeliness  of  the  ECP  process  will  be  based  on  the  findings  regarding 
the  timeliness  of  the  ECP  process  and  on  the  results  of  the  analysis  of 
a  survey  conducted  in  the  SPO  in  the  Fall  of  1989.  The  findings  about 
the  timeliness  of  the  ECP  process  will  be  limited  by  the  time  period 
from  which  the  data  will  be  collected.  Also,  the  findings  from  the 
survey  analysis  will  be  limited  by  the  validity  of  the  survey  instru¬ 
ment.  Rather  than  discuss  the  specific  details  of  these  limitations 
here,  they  will  be  discussed  in  conjunction  with  the  presentation  of  the 
results  in  Chapter  V.  Finally,  although  the  possibility  exists  that  the 
findings  presented  here  are  applicable  to  other  organizations  suffering 
problems  similar  to  those  presented  here,  they  may  not  be  applicable. 
However,  by  performing  the  research  steps  presented  herein,  a  set  of 
problem  solutions  may  be  able  to  be  obtained. 

Definitions 

The  following  words  and  acronyms  are  defined  to  facilitate  the 
mutual  understanding  of  the  words  in  this  thesis.  These  words  are 
broken  out  by  subject  area  and  alphabetized  within  the  subject  area;  for 
example,  acronyms  of  organizations  are  grouped  together,  as  are  the 
group  of  words  applicable  to  the  ECP  process. 
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Of  Organizations.  The  following  acronyms  are  for  the 
organizations  which  are  involved  in  the  ACM  SPO  ECP  process. 


GD/C  --  General  Dynamics/Convair ,  the  prime  contractor  which 
is  responsible  for  producing  ACMs  and  is  the  design  agent  for  the 


ACM 


MDMSC  --  McDonnell  Douglas  Missile  Systems  Company,  the  dual 
source  contractor  responsible  to  producing  "build  to  print"  ACMs 
using  GD/C's  approved  engineering  drawings 

OCALC  --  Oklahoma  City  Air  Logistics  Center,  the  future 
primary  support  organization  for  the  ACM  (10:11) 

SAC  --  Strategic  Air  Command,  the  United  States  Air  Force 
command  which  oversees  all  United  States  strategic  forces  except 
for  submarine  launched  ballistic  missiles  (25:2);  also,  the  future 
user  of  the  ACM  (10:3) 

SPO  --  System  Program  Office,  a  military  organization  which 
is  tasked  with  ensuring  a  weapon  system  is  produced  in  accordance 
with  the  requirements 

Of  the  ECP  Process.  The  following  words  and  acronyms  are  related 
to  the  engineering  change  proposal  process  in  the  ACM  SPO. 

ACSN  --  Advanced  Change  Study  Notification,  "a  document 
prepared  by  [either]  the  contractor  or  the  government  which 
identifies  the  need  for  a  technical  change  to  the  contract.  The 
ACSN  is  used  to  initiate  the  preparation  of  a  routine  ECP"  (7:1). 
ACSN  approval  authorizes  the  contractor  to  expend  the  resources 
necessary  to  prepare  an  ECP  (5). 

CCB  -  Configuration  Control  Board,  a  group  of  individuals 
who  review  EcPs  and  make  a  recommendation,  either  approved, 
approved  with  comments,  disapproved,  or  deferred,  to  the  CCB 
chairman 

CMP  --  Configuration  Management  Plan,  a  document  stipulating 
how  the  contractor  and  the  ACM  SPO  are  to  process  engineering  and 
contract  change  proposals 

ECP  --  Engineering  Change  Proposal,  a  document  used  for 
approval  authority  of  configuration  changes 

TCM  --  Technical  Coordination  Meeting,  a  meeting  between 
government  and  contractor  personnel  to  informally  review  the  TCP 
for  technical  and  administrative  accuracy  before  the  formal  ECP  is 
submitted 
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TCP  --  Technical  Change  Package,  the  complete,  unsigned, 
change  package  which  contains  the  details  of  the  change,  less  cost 
information,  and  which  will  be  corrected  (if  necessary)  based  on 
the  comments  of  the  government  personnel  at  the  TCM  and  submitted 
as  an  ECP 

Of  General  Air  Force  Terms .  The  following  words  and  acronyms  are 
peculiar  to  the  Air  Force  and  include  those  terms  related  to  the 
national  security  of  the  United  States. 

ACM  --  Advanced  Cruise  Missile,  air-launched  air-to-ground 
strategic  nuclear  weapon  systems  which  will  become  a  part  of  the 
United  States'  air-based  nuclear  deterrent  forces  when 
operationally  deployed  (2:2-1) 

Cruise  Missile  --an  unmanned,  winged  air  vehicle  with  a 
nuclear  warhead  that  is  capable  of  guiding  itself  to  a  preselected 
ground  target  (31:156) 

Deliver  --  to  provide;  in  this  thesis,  to  ship  a  missile  to 
the  user  so  that  it  may  be  deployed 

Deploy  --  to  operationally  deploy  a  weapon  is  to  put  it  in  a 
condition  such  that  it  is  ready  to  be  used  for  its  purpose 

Deterrence  --  A  state  of  mind  brought  about  by  a  threat  of 
unacceptable  counteraction  (25:1) 

Deterrent  --a  weapon  system  is  an  effective  deterrent  to 
war  if  it  is  sufficiently  powerful,  in  the  eyes  of  the  enemy,  to 
influence  or  otherwise  prevent  the  enemy  from  engaging  in  hostile 
actions 

Survivability  --to  endure  by  benefiting  fully  from  early 
warning,  quick  reaction,  dispersal,  hardening,  and  mobility  [and 
the  use  of  radar-evading  technology]  (25:1) 

Strategic  --  [as  in  strategic  bombardment]  of  or  relating  to 
the  destruction  of  the  enemy's  war-making  capacity  represented  by 
his  industrial  and  technical  strengths,  most  of  which  are  located 
in  urban  industrial  complexes  (25:1) 

TDY  --  Temporary  Duty,  performance  of  work  at  a  location 
other  than  the  normal  location  for  a  period  of  time  less  than  180 
days 


TRIAD  --  the  land- launched  ICBMs  and  manned  bombers  [with 
their  payloads  of  cruise  missiles  and  bombs]  of  SAC,  combined  with 
the  Navy's  ballistic  missile  submarine  fleet  form  the  TRIAD  of 
strategic  offensive  forces  (25:2) 
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Summary  of  Chapter  I 


In  this  chapter,  general  introductory  background  material  was 
provided  to  describe  the  global  scenario  as  of  the  Spring  of  1990, 
including  the  importance  of  the  Advanced  Cruise  Missile  to  the  continued 
strength  of  the  nation's  deterrent  forces.  The  goal  and  requirements  of 
the  ACM  SPO  were  presented,  and  the  specific  problem  in  the  ACM  SPO  was 
discussed.  One  hypothesis  stated  the  ECP  process  was  untimely,  and  four 
more  hypotheses  were  presented  that  were  based  on  the  first  hypothesis 
being  true.  The  specific  research  topic  was  justified,  and  the  scope 
and  limitations  of  the  research  findings  were  discussed.  Finally,  as  a 
reading  aid,  important  words  and  acronyms  were  defined  and  spelled  out. 
Now  that  the  general  view  has  been  presented,  specific  background 
material  will  be  discussed  in  Chapter  II,  including  the  mission  and 
organization  outside  and  within  the  ACM  SPO,  a  definition  of  the  ECP  and 
a  description  of  the  ECP  process,  and  a  historical  perspective  of  the 
researcher's  prior  activities  in  the  SPO,  including  the  prior  attempts 
at  improving  the  ECP  process  within  the  ACM  SPO. 
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II .  Background 


Chapter  I  presented  specific  hypotheses  about  the  timeliness  of 
the  ECP  process  and  the  ability  of  the  SPO  to  effectively  perform  the 
ECP  process.  To  be  able  to  better  understand  the  hypotheses,  this 
chapter  describes  the  ACM  SPO,  including  its  mission  and  its  organiza¬ 
tional  structure.  The  ACM  SPO  works  with  outside  agencies,  so  these 
agencies  will  be  presented  as  well.  The  ACM  SPO  and  its  relationship  to 
other  organizations  will  facilitate  the  discussion  of  the  ECP  and 
especially  the  ECP  process.  Next,  the  time  requirements  within  which 
ECPs  should  be  implemented  are  presented.  Finally,  the  researcher 
worked  in  the  ACM  SPO  from  the  Fall  of  1986  to  the  Summer  of  1989,  so 
his  role  in  the  SPO  will  be  presented  along  with  the  prior  attempts  of 
the  SPO  to  correct  ECP  process  problems. 


ACM  SPO.  Mission  and  Structure 

So  that  the  reader  may  better  understand  the  environment  of  the 
ACM  SPO,  the  following  discussion  restates  the  ACM  SPO's  specific 
mission,  explains  how  the  ACM  SPO  is  organized  within  the  Air  Force  and 
within  itself,  and  shows  the  approximate  number  of  people  in  the  SPO  who 
are  actually  involved  in  the  ECP  process. 

The  ACM  SPO  Mission.  The  mission  of  the  ACM  SPO  is  similar  to  the 
mission  of  the  typical  SPO: 

The  Program  Office  (PO)  [also,  System  Program  Office  or  SPO]  is 
the  "backbone"  of  the  system  acquisition  process.  Without  it,  the 
program  will  never  be  able  to  stand  successfully.  The  PO  is  the 
management  focal  point  . . .  for  all  agencies  involved  in  the  system 
acquisition  as  well  as  the  only  organization  authorized  to  direct 
the  system  contractor's  efforts.  (19:1) 
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With  this  kind  of  responsibility,  the  general  mission  of  the  typical  SPO 
is  "to  get  a  system  to  the  user  which  meets  cost,  schedule,  logistic 
supportability ,  and  performance  requirements  levied  by  DOD”  (19:3). 

Given  this  general  mission,  the  ACM  SPO's  specific  mission  is  to 
deliver  Advanced  Cruise  Missiles  and  the  necessary  support  equipment  to 
SAC,  meeting  the  cost,  schedule,  logistic  supportability,  and 
performance  requirements  outlined  in  its  Program  Management  Directive 
(10:1-8)  . 

External  Hierarchy.  Before  discussing  the  ACM  SPO  in  detail,  a 
brief  synopsis  of  the  chain  of  command  from  the  United  States  Department 
of  Defense  (DOD)  to  the  ACM  SPO  is  in  order.  Figure  4  shows  the 
external  hierarchy  from  the  Department  of  Defense  to  the  ACM  SPO. 

Within  the  DOD,  the  United  States  Air  Force  (USAF)  is  responsible  for 
the  acquisition,  support,  and  operations  of  all  USAF  weapon  systems. 
Examples  of  USAF  weapon  systems  include  tactical,  airlift,  strategic 
bombing,  and  reconnaissance  aircraft  and  systems  which  are  intended  to 
support,  respectively,  Army  ground  troops,  worldwide  mobility,  offensive 
and  counteroffensive  attacks,  and  surveillance  (28:1-2  and  24:1-2). 
Another  example  includes  air-launched  strategic  nuclear  missiles  which 
are  used  primarily  for  deterrence  to  war  (31:156).  Within  the  USAF, 
departments  or  commands  are  broken  out  primarily  by  the  function  they 
serve  within  the  USAF.  For  example,  the  Strategic  Air  Command  (SAC)  is 
responsible  for  the  operation  of  all  strategic  weapons  systems, 
including  the  strategic  bombing  and  reconnaissance  aircraft  and  the 
strategic  missiles  (25:3).  Air  Force  Systems  Command  (AFSC)  is  another 
USAF  command;  AFSC  is  responsible  for  overseeing  the  development  and 
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Figure  4.  Chain  of  Command  from  DoD  to  the  ACM  SPO 


procurement  of  practically  all  USAF  weapons,  regardless  of  the  command 
which  will  ultimately  use  them  (27:2).  AFSC  is  divided  into  divisions 
which  either  concentrate  primarily  on  products  or  on  research.  The 
Aeronautical  Systems  Division  (ASD) ,  located  at  Wright  -  Patterson  AFB  in 
Ohio,  is  one  of  those  product  divisions.  ASD  has  SPOs  which  are 
responsible  for  the  design,  development,  production,  and  initial  opera¬ 
tional  support  of  a  specific  weapon  system  or  group  of  weapon  systems. 
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Whereas  the  ACM,  F-16,  B-l,  and  B-2  SPOs  are  concerned  with  all  aspects 
of  one  specific  weapon  system,  the  Training  SPO  is  an  example  of  a  SPO 
which  deals  with  a  group  of  weapon  systems.  ASD  SPOs  usually  have  a 
two- letter  designation  that  is  used  as  an  address.  The  ACM  SPO's  two- 
letter  symbol  is  "VC,"  so  that  a  reference  to  ASD/VC  would  indicate  the 
ACM  SPO  within  Aeronautical  Systems  Division.  With  approximately  140 
government  personnel  and  about  40  additional  contractor  personnel,  the 
ACM  SPO  also  has  within  itself  a  logically- arranged  organizational 
structure;  this  internal  structure  is  described  next. 

ACM  SPO  Internal  Organization.  The  typical  SPO  is  organized  by 
the  program  director,  who  "purposely  tailors  the  internal  organiza¬ 
tional  configuration  of  the  SPO  to  fit  the  needs  and  constraints  of  the 
particular  program"  (19:1).  The  ACM  SPO  is  currently  organized  as  shown 
in  Figure  5,  which  indicates  the  structure  and  the  number  of  authorized 
and  actual  personnel  in  each  directorate  as  of  June  8,  1990;  these 
numbers  have  remained  relatively  constant  through  the  period  in  which 
this  thesis  was  performed  (15).  The  program  director  is  entitled  to 
organize  the  SPO  because  "he  must  define  the  objectives  and  make  the 
ultimate  decisions  which  affect  the  cost,  schedule,  logistics  suppor- 
tability,  and  performance  of  the  system"  (19:1).  The  ACM  SPO  director 
has  divided  his  organization  into  13  directorates ;  each  of  these 
directorates  performs  specific  roles  or  functions  in  the  SPO,  as 
outlined  below  (their  office  symbols  are  indicated  in  parentheses)  (16). 

Projects  &  Dual  Source  Directorates  (VCA  &  VCB) .  The 
Projects  Directorate  consists  of  the  project  managers  who  are 
responsible  for  the  acquisition  and  integration  of  specific  parts  of  the 
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Note:  Functional  Organizations  Designated  by  Double  Lines 


missile  into  the  whole  (16).  For  example,  the  development,  production, 
and  integration  of  the  missile's  engine,  guidance  system,  software, 
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etc.,  must  be  managed  so  that  the  end  result  is  a  functioning  cruise 
missile  (16).  The  Dual  Source  Directorate  (VCB)  is  essentially 
identical  in  purpose  and  function  to  VCA,  except  that  VCB  oversees  the 
activities  of  the  dual  source  contractor,  McDonnell  Douglas  Missile 
Systems  Company  (MDMSC) ,  while  VCA  oversees  the  General  Dynamics/Convair 
(GD/C)  effort.  These  project  managers  help  the  program  director  fulfill 
the  SPO's  mission,  and  they  must  get  work  done  with  the  help  of 
personnel  from  the  other  directorates,  including  engineering,  manufac¬ 
turing,  contracts,  program  control,  and  the  rest  shown  in  Figure  5  (16). 

These  other  directorates  are  designated  as  "functional" 
directorates  because  their  primary  mission  usually  consists  of  advising 
and  supporting  the  program  director  in  their  specific  "functional" 
specialties  (19:10).  Of  these  functional  directorates,  the  most 
important  one  to  the  processing  of  Engineering  Change  Proposals  is  the 
Configuration  Management  Directorate  (VCC) .  The  purpose  of  the  other 
functional  personnel,  with  respect  to  the  ECP  process,  is  usually  to 
review  the  ECP  and  provide  comments  regarding  it  relative  to  their 
specific  functional  area  (7:7).  In  light  of  this,  VCC's  role  in  the  ACM 
SPO  is  presented  below,  while  the  specific  roles  of  the  other  functional 
organizations  are  presented  in  Appendix  A. 

Configuration  Management  Directorate  (VCC) .  Generally 

speaking , 

Configuration  management  is  charged  with  formalizing  the  system 
requirements  into  systems  specifications,  controlling  the 
hardware/software  configuration,  and  accounting  for  all  configura¬ 
tion  items.  In  addition  to  performing  the  data  management 
function  for  the  SPO,  this  [directorate]  manages  the  system 
configuration  control  board  [CCB]  activities  and  all  engineering 
change  proposals  [ ECPs ] .  (19:12) 
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Specifically,  VCC  is  responsible  for  the  following  activities  related  to 
the  ECP  process  in  the  ACM  SPO: 

-  Receiving  and  evaluating  all  change  proposals  [and]  ACSNs  ... 
for  compliance  with  [the  requirements], 

-  Establishing  and  maintaining  official  files  for  all  proposals. 

-  Establishing  and  maintaining  a  tracking  system  for  all  change 
proposals/ACSNs . 

-  Identifying  the  OPR  for  each  change . 

-  Distributing  proposals  to  all  CCB  members  for  review  and 
comment . 

-  Preparing  and  distributing  a  periodic  report  identifying  the 
current  status  of  all  open  change  proposals. 

-  Initiating  a  bi-monthly  telecon  with  the  contractor  to 
establish/maintain  change  proposal  schedules. 

-  Initiating  and  chairing  TCM  meetings 

-  Preparing  and  distributing  CCB  membership  orders. 

-  Preparing  and  distributing  the  CCB  agenda  3  working  days  before 
each  CCB  meeting. 

-  Serving  as  Secretariat  to  the  CCB. 

-  Preparing  CCB  Directives  (CCBDs)  and  obtaining  required 
s ignatures . 

-  Preparing  and  distributing  CCB  minutes  not  later  than  3  working 
days  after  CCB  meeting. 

-  Ensuring  all  companion  change  proposals  are  sent  to  the 
appropriate  proposal  manager  for  processing  with  the  original 
change  proposal. 

-  Provide  all  unsolicited  changes/revisions  to  previously 
dispositioned  change  proposals  to  the  OPR  for  resolution.  (7:4) 

Many  of  the  terms  above  have  not  yet  been  defined  but  instead  are 

defined  in  conjunction  with  the  description  of  the  ECP  process.  Suffice 

it  to  say  that  the  role  VCC  plays  in  the  ECP  process  is  crucial  to  its 

success  in  implementing  design  changes  in  a  timely  and  efficient  manner. 
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ACM  SPO  Personnel  Involved  in  the  ECP  Process.  Within  each  of  the 


directorates  in  Figure  5,  a  number  of  individuals  get  involved  with 
processing  an  ECP.  Table  1  shows  the  approximate  number  of  people  in 
each  directorate  who  may  be  involved  in  the  ECP  process  at  any  one  time 
(17).  As  stated  previously,  the  roles  of  the  individuals  in  these 


TABLE  1 

NUMBER  OF  ACM  SPO  PERSONNEL  INVOLVED  IN  THE  ECP  PROCESS  (17) 


Directorate 

Number  of 

Personnel 

Proj  ects 

9 

Dual  Source 

4 

Configuration  and  Data  Mgmt 

8 

Manufacturing 

5 

Engineering 

17 

Safety 

1 

Contracts 

8 

Logistics 

5 

Program  Control 

3 

Security 

3 

SAC  Liaison 

1 

Test 

3 

SEMCO ,  ARINC,  and  TASK 

15 

Total 

82 

Total  in  the  ACM  SPO 

182 

organizations  with  respect  to  the  ECP  process  is  primarily  to  review  the 
ECPs  with  respect  to  their  specific  functional  specialty  (7:7). 

Summary,  ACM  SPO  Organization.  Here  we've  seen  the  specific 
mission  and  organization  structure  of  the  ACM  SPO.  We  looked  at  how  the 
ACM  SPO  is  organized  with  respect  to  the  Department  of  Defense  and  how 
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the  SPO  has  organized  itself  to  accomplish  its  mission.  Furthermore, 
we've  seen  how  many  people  in  the  SPO  get  involved  in  the  ECP  process. 
However,  more  people  are  involved  in  the  ACM  SPO's  ECP  process  than  just 
the  ACM  SPO  personnel.  This  next  section  discusses  the  other  players  in 
the  ECP  process . 

Outside  Organizations  that  Work  with  the  ACM  SPO 

The  following  organizations  work  with  ACM  SPO  personnel  on  at 
least  an  occasional  basis,  based  on  whether  or  not  the  specific  ECP  in 
question  impacts  their  areas  of  responsibility.  Figure  6  provides  a 
picture  of  the  relationships  that  exist  between  these  organizations. 

The  double  lines  represent  direct  contractual  arrangements  between 
organizations,  while  the  single  line  indicates  the  lines  of 
communication  and  coordination  between  organizations  that  do  not  have 
contractual  agreements. 

General  Dvnamics/Convair .  GD/C  is  one  of  the  ACM  SPO's  prime 
contractors,  prime  in  the  sense  that  it  has  signed  a  contractual 
agreement  with  the  SPO  to  deliver  complete  and  functioning  weapon  sys¬ 
tems  (16).  The  prime  contractor  can  choose  to  build  what  the  government 
wants  by  itself,  or  it  can  sub-contract  out  some  or  all  of  the  pieces  of 
the  effort  to  other  organizations.  GD/C  has  chosen  to  sub-contract  out 
parts  of  the  effort,  including  the  navigation,  fin  control  actuators, 
and  other  systems  to  others  while  retaining  overall  integration  respon¬ 
sibility  for  itself  (16).  GD/C  was  chosen  in  1983  to  develop  the  ACM, 
which  it  has  done  (16).  Although  GD/C  has  produced  and  delivered  some 
ACMs  to  the  Air  Force,  it  is  still  involved  in  the  modification  of  the 
design  to  some  extent,  especially  in  the  area  of  engineering  design 
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Figure  6.  Organizations  that  Interact  With  the  ACM  SPO 


changes  and  modifications  (16).  GD/C  must  interface  directly  with  MDMSC 
before  it  can  propose  an  engineering  design  change  because  all  design 
changes  that  GD/C  may  wish  to  propose  to  the  government  must  first  be 
analyzed  by  MDMSC  before  they  can  be  proposed  to  the  government  (this  is 
true  because  of  the  dual  source  relationship  of  MDMSC  to  GD/C,  described 
below)  (6:13).  Furthermore,  GD/C  interfaces  with  Williams  International 
(WI),  the  manufacturer  of  the  engine,  via  an  Interface  Memorandum  (IFM) 
when  GD/C  experiences  difficulty  integrating  the  engine  into  the  missile 
(24).  GD/C  is  responsible  for  integrating  the  engine  into  the  missile, 
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so  IFMs  are  used  as  an  informal  mode  of  communication  between  the  two 
even  though  a  contractual  relationship  does  not  exist  between  them  (24) . 

McDonnell  Douglas  Missile  Systems  Company.  MDMSC  is  the  second 
prime  contractor  building  ACMs .  MDMSC  is  what  is  known  as  a  "dual 
source"  contractor  which  means  that  it  is  responsible  to  the  government 
for  producing  "build  to  print"  hardware;  MDMSC  will  use  the  same  blue 
prints  that  GD/C  uses  and  produce  hardware  that  is  indistinguishable 
from  the  hardware  built  by  GD/C  (18).  MDMSC's  contract  is  with  the  ACM 
SPO  (18).  Furthermore,  MDMSC  must  work  with  GD/C  on  ECPs  for  the  same 
reason  that  GD/C  must  work  with  MDMSC:  the  dual  source  relationship 
means  that  any  design  changes  that  either  GD/C  or  MDMSC  wishes  to 
propose  must  be  analyzed  by  the  other  contractor  before  they  can  be 
proposed  to  the  government  (6:13).  Because  of  this  requirement,  both 
contractors  share  in  the  same  knowledge  throughout  the  dual  source 
arrangement.  MDMSC  was  chosen,  in  a  competitive  source  selection,  to 
fulfil  the  role  as  the  second  producer  of  ACMs  in  November,  1987  (18). 

It  is  important  for  MDMSC  to  be  in  the  ECP  process  loop  not  only  because 
of  the  DOD-STD-480A  requirement  but  also  because  any  approved  design 
changes  that  GD/C  implements  in  its  hardware  or  software,  depending  on 
the  specific  change,  may  also  be  required  to  be  implemented  in  the 
hardware  or  software  of  the  missiles  that  MDMSC  builds  (18). 

Furthermore,  MDMSC  was  chosen  to  not  only  produce  ACMs  "build  to  print," 
but  also  to  propose  design  changes  when  it  can  show  that  cost  savings, 
maintainability  enhancements,  or  other  benefits  to  the  government  can 
possibly  be  achieved  (18).  However,  potential  design  changes  that  could 
be  beneficial  to  the  government  may  not  be  proposed  by  these  contractors 
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because  of  the  adversarial  relationship  that  potentially  exists  between 
them  (18) .  This  adversarial  relationship  could  exist  based  on  -he 
following  facts: 

1.  General  Dynamics  spent  more  money  designing,  developing,  and 
testing  the  ACM  than  it  received  from  the  government;  this  money  can 
only  be  recouped  by  obtaining  future  production  contracts  and  effi¬ 
ciently  delivering  systerj  to  those  contracts  (18). 

2.  GD/C,  before  the  dual  source  source  selection  was  initiated, 
was  hoping  to  have  a  chance  to  recoup  some  of  its  losses  on  future 
production  contracts  (18).  However,  the  dual  source  is  expected  to 
compete  for  a  lion's  share,  perhaps  up  to  70%,  of  future  production 
efforts,  so  GD/C's  chances  of  recouping  lost  dollars  is  diminished  (18). 

3.  After  MDMSC  was  chosen,  GD/C  was  left  with  the  realization 
that  not  only  was  MDMSC  the  new  competitor  for  future  ACM  production 
contracts,  but  also  that  this  was  the  third  weapon  system  that  GD/C  lied 
designed  but  then  been  second  sourced  by  MDMSC- -the  first  two  systems 
were  the  Tomahawk  Cruise  Missile  and  the  Ground- Launched  Cruise  Missile 
(GLCM)  (18).  Furthermore,  for  the  Tomahawk  and  GLCM  programs,  MDMSC  was 
the  contractor  that  built  the  guidance  system  for  GD/C,  and  GD/C  inte¬ 
grated  the  guidance  unit  into  the  structure  of  the  missiles  (18).  So, 
when  MDMSC  became  the  second  source  to  GD/C,  MDMSC  gained  GD/C's  experi¬ 
ence  and  technology  for  building  the  structure  of  cruise  missiles  and 
GD/C  gained  MDMSC' s  experience  and  technology  for  building  guidance  sys¬ 
tems  (18).  However,  since  MDMSC  had  not  been  involved  in  building  any 
of  the  ACM's  subsystems,  MDMSC  had  nothing  to  offer  GD/C  with  respect  to 
the  ACM,  except  to  take  future  dollars  away  from  General  Dynamics  (18) . 
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Because  of  these  events,  neither  GD/C  nor  MDMSC  is  totally 
committed  to  sharing  knowledge  gained  through  their  particular 
experiences,  and,  as  a  result,  engineering  change  proposals  that  could 
save  the  government  money  or  make  the  maintainers'  lives  easier  may  be 
delayed  or  withheld  by  one  contractor  because  of  the  potential  benefits 
the  other  contractor  could  reap  as  a  result  of  the  idea  (18). 

The  Engine  SPO  (YZ) .  The  Engine  SPO  has  personnel  in  it  who  are 
the  procuring  agents  for  the  engine  that  is  installed  in  the  ACM  (24) . 
The  YZ  SPO  is  a  system  program  office  in  Aeronautical  Systems  Division, 
and  the  contract  for  producing  the  ACM  engines  is  between  YZ  and  WI 
(24).  Whenever  an  IFM  is  sent  between  GD/C  and  WI ,  both  YZ  and  the  ACM 
SPO  receive  information  copies  of  the  message  (24).  The  engine  project 
officer  in  the  ACM  SPO  then  works  with  YZ  and  GD/C  personnel  to  resolve 
any  problems  or  issues  (24) . 

Defense  Contract  Administration  Service.  DCAS  is  a  government 
organization  which  provides  the  ACM  SPO  with  administrative  and 
technical  support  (16).  They  work  alongside  GD/C  personnel,  and  have 
their  office  in  the  same  building  in  which  the  ACM  is  assembled.  DCAS 
provides  valuable  experience  to  the  ACM  SPO,  especially  in  the  area  of 
how  GD/C  operates  (16).  With  respect  to  the  ECP  process,  they  review 
and  comment  on  all  ECPs . 

Oklahoma  City  Air  Logistics  Center.  OCALC  is  the  organization 
which  will  become  the  primary  supporter  of  the  ACM  after  a  milestone 
event  called  Program  Management  Responsibility  Transfer  (PMRT)  (10:11). 
While  the  ACM  design  is  still  undergoing  modifications,  the  ACM  SPO  is 
responsible  for  supporting  the  fielded  weapon  systems  (10:4).  However, 
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after  the  design  is  stabilized,  PMRT  will  take  place,  with  OCALC's 
concurrence,  and  OCALC  will  become  responsible  for  all  support  aspects 
of  the  program  (10:11-12).  Thus  OCALC  must  be  involved  in  the  ECP 
process  so  that  it  will  know  not  only  the  configuration  of  the  missiles 
that  it  is  becoming  responsible  for,  but  also  how  to  support  and 
maintain  them. 

Strategic  Air  Command.  SAC  is  the  ultimate  user  of  the  Advanced 

Cruise  Missile  (10:3).  As  the  user,  SAC  will  perform  maintenance  on  the 

missiles,  so  it  is  very  important  for  SAC  maintenance  personnel  to  be 

able  to  determine  each  missile's  configuration  (10:3).  SAC  is  involved 

in  the  ECP  process  through  their  in-the-SPO  resource,  VCS  (16).  This 

office  is  manned  by  a  Strategic  Air  Command  mid- level  liaison  officer 

who  is  invited  to  review  all  ECPs  and  attend  all  CCBs  (16).  This 

officer  provides  the  SPO  the  user's  perspective,  and  vice  versa. 

SAC  Liaison  Office  (VCS).  VCS  is  not  shown  on  Figure  5,  even 

though  its  office  is  in  the  ACM  SPO,  because  the  office  is  manned  by  a 

SAC  resource  (16).  Thus  this  officer  reports  only  to  SAC,  not  the  SPO, 

allowing  the  using  command  (SAC)  to  have  a  direct  advocate  in  the  SPO 

(16).  As  SAC  spokesmen,  VCS  personnel 

monitor  and  support  system  activities  involving  the  system  design, 
development,  test,  production,  performance,  reliability,  main¬ 
tainability,  training,  and  deployment.  They  advise  the  PM  of  the 
operating  command's  interests  and  concerns  related  to  the  opera¬ 
tional  use  of  the  system.  (19:13) 

B-52  Integration  and  B-1B  Integration.  These  two  offices  also  are 
manned  by  personnel  from  other  Aeronautical  Systems  Division  personnel 
who  oversee  the  integration  effort  between  the  ACM  and  their  respective 
carrier  aircraft. 
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Other  Organizations,  Other  organizations  occasionally  get 


involved  with  ACM  SPO  ECPs ,  but  mainly  for  support  and  other  peripheral 
equipment  (5).  These  organizations  will  not  be  addressed  here,  in  the 
interests  of  time  and  space.  We've  now  seen  all  of  the  personnel  and 
organizations  which  get  involved  with  the  ECP  process;  it  is  now  time  to 
see  just  exactly  what  the  ECP  is  and  how  one  gets  processed  in  the  ACM 
SPO. 

Engineering  Change  Proposal:  Purpose  and  Process 

Now  that  all  of  the  players  in  the  ECP  process  have  been 
introduced,  the  ECP  process  and  the  personnel  who  perform  the  specific 
steps  in  the  process  can  be  described.  This  section  will  define  and 
describe  the  engineering  change  proposal's  purpose  and  its  general 
process  flow.  Furthermore,  the  specific  ECP  process  flow  in  the  ACM  SPO 
will  be  addressed,  since  its  flow  varies  from  the  flow  described  in  the 
regulations.  This  discussion  will  lend  itself  to  the  discussion  in  the 
next  section,  which  will  include  a  discussion  of  the  time  requirements 
within  which  the  ACM  SPO  ECP  process  is  to  be  performed. 

ECP  Purpose .  An  ECP  is  a  document  designed  to  maintain  configura¬ 
tion  control  over  hardware  and  software  being  produced  for  the  govern¬ 
ment  (6:iii).  Department  of  Defense  Military  Standard  480A  (DOD-STD- 
480A) ,  Military  Standard  for  Configuration  Control  Engineering  Changes, 
Deviations,  and  Waivers,  stipulates  that  proposed  configuration  changes 
will  be  identified  via  an  engineering  change  proposal  (6:1).  DOD-STD- 
480A  provides  the  information  needed  to  ensure  contractors  maintain 
configuration  control  over  the  hardware  and  software  they  produce  for 
the  government  (6:1).  Specifically,  ACM  VC  Operating  Instruction  800-1 


32 


(VC  01  800-1),  ACM  Configuration  Contract  Change  Control,  echoes  the 

requirements  for  ECPs  as  stipulated  in  DOD-STD-480A: 

Each  proposed  change  in  configuration  will  identify  the  necessary 
corresponding  changes  and/or  additions  in  related  support  equip¬ 
ment,  tooling,  spares,  training  equipment,  technical  publications, 
associate  and/or  dual  source  contractors,  and  any  other  areas 
needed  to  ensure  total  weapon  system  compatibility.  When  a 
proposed  change  requires  additional  tests,  the  scope  of  the  test 
shall  be  defined. . . . 

Each  proposed  change  shall  identify  the  total  impact  to  the 
program,  if  approved,  including  the  effect  of  the  change  on 
development,  production,  and  retrofit  to  the  system  or  equipment. 
Proposed  change  impacts  on  the  dual  source  contractor  shall  be 
included  in  the  decision  process.  (7:3) 

Configuration  control  is  critical  to  the  government  because  without  it 

the  systems  will  not  be  able  to  be  maintained  effectively  once  they  are 

deployed  (17).  The  military  standard  stipulates  that,  "unless  otherwise 

specified  by  the  procuring  activity  [in  this  case  the  ACM  SPO] ,  receipt 

of  contractual  approval  shall  constitute  the  sole  authority  for  the 

contractor  to  effect  the  change"  (6:15).  In  summary,  engineering  change 

proposals  have  a  specific  purpose,  to  maintain  configuration  control, 

and  they  must  be  used  for  approval  of  configuration  changes  (6:15). 

Without  the  government's  contractual  approval,  the  contractor  is  not 

authorized  to  perform  the  work  necessary  to  implement  a  change,  so 

delays  in  ECP  approval  can  result  and  have  resulted  in  direct  delays  to 

scheduled  missile  deliveries  (17). 

ECP  Process,  by  Regulation.  With  the  definition  and  purpose  of 

the  ECP  still  in  mind,  what  are  the  necessary  steps  to  obtain  approval 

to  implement  a  configuration  change?  DOD-STD-480A  states  the  general 

steps  as  follows: 

4.1  General .  The  steps  in  processing  an  engineering  change  con¬ 
sist  of  the  following:  (a)  determination  of  a  need  for  the 
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change,  (b)  establishment  by  the  originator  of  a  classification  of 
the  engineering  change  as  Class  I  or  Class  II,  (c)  preparation  of 
an  ECP,  (d)  submittal  to  the  Government,  (e)  review,  (f) 
approval/disapproval  or  concurrence/nonconcurrence  in  classifica¬ 
tion,  and  (g)  incorporation  of  approved  (or  concurred  in) 
engineering  changes  in  the  configuration  item  and  in  the  data, 
including  when  applicable,  negotiation  into  the  contract.  (6:3) 

Although  DOD-STD-480A  lists  the  general  steps  involved  in  the  ECP 
process,  the  ACM  SPO  has  additional  steps  over  and  above  those  listed 
above.  Indeed,  the  ACM  SPO  has  not  only  added  two  steps  into  the  proc¬ 
ess,  but  also  often  recycles  ECPs  through  steps  c,  d,  e,  and  f  more  than 
once  because  of  problems  with  the  words  or  content  of  the  ECP  packages. 
These  problems  require  the  contractor  to  make  changes  to  the  document 
and  resubmit  it  to  the  government  (17).  Because  of  these  extra  steps, 
the  ACM  SPO  process  as  it  exists  in  practice  will  now  be  discussed. 

The  ACM  SPO  ECP  Process,  in  Practice.  Figure  7  shows  the  overall 
flow  of  the  ECP  process  in  the  ACM  SPO.  The  boxes  with  double  lines 
around  them  indicate  those  steps  which  are  not  included  in  the 
regulation  but  instead  have  been  added  to  the  process  either  in  an 
attempt  to  improve  the  overall  timeliness  of  the  process  or  because  the 
process  is  not  functioning  ideally.  The  discussion  below  will  follow 
the  path  of  the  ECP  from  the  determination  of  a  need  for  a  change  to  the 
supplemental  agreement  (S/A)  step,  describing  each  step  along  the  way. 

Determine  a  Need  for  a  Change.  The  first  step  in  processing 
an  engineering  change  is  to  determine  a  need  for  a  change  (6:3).  Any 
organization,  government  or  contractor,  can  initiate  the  change  process 
by  identifying  a  deficiency  between  how  the  system  works  and  how  it  is 
supposed  to,  or  should,  work.  The  contractor,  for  example,  could  initi¬ 
ate  the  change  process  by  discovering,  through  testing,  a  recurrent 
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Figure  7.  ECP  Flow  in  the  ACM  SPO 


problem  that  has  no  apparent  or  immediate  solution.  If  it  would  be  more 
cost  effective  for  the  contractor  to  identify  and  fix  the  recurring 
problem,  rather  than  throw  away  the  non-functioning  hardware,  it  would 
probably  initiate  a  study  to  find  the  source  of  the  problem.  Such  a 
study  may  reveal  a  problem  in  a  hardware  design.  To  solve  the  problem, 
the  contractor  may  decide  to  change  the  design.  If  the  problem  results 
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in  a  deficiency  to  the  contract,  then  the  contractor  might  prepare  an 
ECP  and  submit  it  to  the  government.  However,  if  the  solution  to  the 
problem  is  beyond  the  scope  of  the  current  contract,  then  the  contractor 

may  submit  an  Advanced  Change  Study  Notice  (ACSN) ,  a  standard  government 

form  used  for  these  types  of  requests,  to  the  government  (17).  The  ACSN 
will  identify  the  general  concept  of  the  change,  the  potential  benefits 
of  the  change,  and  the  rough  order  of  magnitude  cost  of  the  change  (17). 
The  ACSN  allows  the  contractor  to  propose  the  idea  and,  if  the  govern¬ 
ment  likes  it,  the  contractor  may  be  asked  to  expend  the  resources  to 

prepare  the  ECP  that  includes  not  only  the  technical  details  of  the 

change  but  also  the  proposed  costs  of  implementing  the  change  (17).  An 
ACSN  can  also  be  initiated  by  the  government;  the  ACSN  could  direct  the 
contractor  to  prepare  an  ECP  based  on  the  information  in  the  ACSN.  Of 
course,  this  ACSN  would  also  have  to  authorize  the  contractor  to  spend 
the  resources  necessary  to  prepare  the  change  (17).  Based  on  the  con¬ 
tractor's  response  as  documented  in  the  ECP,  the  government  may  decide 
to  disapprove  the  change,  but  would  still  be  required  to  pay  the 
contractor  for  the  costs  incurred  in  preparing  the  ECP  (17).  These  are 
just  examples  of  how  a  design  change  gets  initiated- -  regardless  of  how 
one  is  initiated,  however,  the  remaining  steps  leading  to  actual 
implementation  of  the  design  change  follow  the  same  general  flow  (17)  . 

Establish  the  Classification.  Once  a  need  for  a  design 
change  is  identified,  it  must  be  classified  as  either  a  Class  1  or  a 
Class  II  change  (6:3).  Basically,  the  classification  can  be  made  by 
following  the  guidelines  listed  below: 
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4.2.1  Class  I  engineering  change.  An  engineering  change  shall  be 
classified  Class  I  when  one  or  more  of  the  factors  listed  (sub- 
paragraphs  a,  b,  c,  d,  or  e)  below  is  affected: 

a.  the  functional  or  allocated  configuration  identification... 

b.  The  product  configuration  identification  (PCI)  as  contrac¬ 
tually  specified  (or  as  applied  to  government  activities) , 
excluding  referenced  drawings,  specifications,  listings  of 
computer  program  instructions,  and  actual  data  values 

Note:  In  the  above  definition  of  a  Class  I  engineering 

change,  the  words  "excluding  referenced  drawings,  specifications, 
listing  of  computer  program  instructions,  and  actual  data  values" 
in  4.2.1b  shall  not  be  interpreted  to  exclude  these  items 
prescribed  directly  in  a  contract  to  define  contract  line  items. 
Other  drawings,  specifications,  computer  program  instructions,  and 
actual  data  values,  whether  referenced  in  documents  or  listed  on 
associated  lists,  are  excluded  from  4.2.1b  but  included  in  4.2.1c, 
d,  and  e. 


c.  Technical  requirements  below  contained  in  the  PCI  as  contrac¬ 
tually  specified,  including  referenced  drawings  and  specifications 


(1)  Performance  outside  stated  tolerance 

(2)  Reliability,  maintainability,  or  survivability  outside 
stated  tolerance 

(3)  Weight,  balance,  moment  of  inertia 

(4)  Interface  characteristics 


d.  Non- technical  contractual  provisions 


(1)  Fee 

(2)  Incentives 

(3)  Cost  to  the  Government 

(4)  Schedules 

(5)  Guarantees  or  deliveries 


e.  Other  factors 


(1)  Government  furnished  equipment  (GFE) 

(2)  Safety 

(3)  Electromagnetic  characteristics 

(4)  Operational,  test,  or  maintenance  computer  programs 

(5)  Compatibility  with  support  equipment,  trainers,  or 
training  devices/equipment 

(6)  Configuration  to  the  extent  that  retrofit  action  would 
be  taken 

(7)  Delivered  operation  and  maintenance  manuals  for  which 
adequate  change/revision  funding  is  not  on  existing  contracts 
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(8)  Pre-set  adjustments  or  schedules  affecting  operating 
limits  or  performance  to  such  extent  as  to  require  assignment  of  a 
new  identification  number 

(9)  Interchangeability,  substitutability,  or 
replaceability ,  as  applied  to  CIs,  and  to  all  subassemblies  and 
parts  of  reparable  CIs,  excluding  the  pieces  and  parts  of  non¬ 
reparable  subassemblies 

(10)  Sources  of  CIs  or  reparable  items  at  any  level  defined 
by  source  control  drawings 

(11)  Skills,  manning,  training,  biomedical  factors,  or  human 
engineering  design  (6:3-5) 

The  above  factors,  for  ease  of  discussion,  will  be  referred  to  simply  as 
"form,  fit,  and  function"  factors  throughout  the  remainder  of  this 
document.  So,  if  the  form,  fit,  or  function  will  be  affected,  then  the 
change  is  considered  a  Class  I  change.  Conversely,  if  form,  fit,  or 
function  is  not  affected,  then  the  change  is  considered  a  Class  II 
change  (6:5).  DOD-STD-480A  stipulates  that  the  originator  of  the  change 
determines  its  classification;  however,  the  military  standard  also 
indicates  that  the  government  has  the  final  approval  authority  with 
respect  to  the  classification  (6:3). 

Why  the  Distinction  Between  the  Classes?  The 
distinction  between  the  classes  is  made  to  identify  the  scope  of  the 
change  and  for  administrative  purposes  - -ACM  Class  I  changes  must  be  ap¬ 
proved  by  the  government  using  the  ACM  SPO  ECP  process,  while  Class  II 
changes  only  require  government  concurrence  that  the  change  is  indeed  a 
Class  II  change  (17).  In  fact,  the  ACM  SPO  has  delegated  the  Class  II 
concurrence  authority  to  DCAS  (17),  Thus,  when  the  contractor  submits  a 
Class  II  engineering  change  proposal  to  DCAS  for  concurrence,  DCAS 
verifies  the  form,  fit,  and  function  are  not  affected  before  it  concurs 
with  the  classification  (17).  DCAS  can  disagree,  and  has  disagreed, 
with  the  contractor's  submittal  as  a  Class  II  change.  When  DCAS  does 
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this,  it  refers  the  change  to  the  ACM  SPO  which  then  can  either  concur 
with  the  classification  as  submitted  or  require  the  contractor  to  submit 
the  proposal  as  a  Class  I  change  through  the  SPO  ECP  process  (17). 
Furthermore,  GD/C  and  MDMSC  must  agree  upon  the  classification,  other¬ 
wise  the  change  will  be  referred  to  the  SPO  (17).  DCAS  will  know  about 
the  lack  of  consensus  on  classification  of  the  ECP  because,  before  an 
ECP  can  be  submitted  to  the  government,  the  originator  of  the  change 
must  coordinate  the  proposal  with  the  other  prime  contractor  and 
indicate  on  the  ECP  the  results  of  the  coordination  effort  (6:13).  The 
distinction  between  the  classes  is  also  made  to  ensure  the  proper 
program  concerns  are  addressed,  including,  for  example,  whether  or  not 
logistics  will  be  impacted  because  of  the  design  change  (17).  For 
example,  a  change  to  a  card  inside  an  electronic  box  may  not  change  the 
physical  form  of  the  electronic  box,  its  fit  within  the  missile,  or  its 
function;  however,  if  the  change  results  in  a  different  part  that  must 
be  spared  or  otherwise  accounted  for,  then  the  change  will  need  to  be 
classified  as  a  Class  I  change  (17).  For  those  changes  identified  as 
Class  I,  the  next  step  in  the  ECP  SPO  process  is  to  prepare  and  submit  a 
TCP  to  the  government. 

Prepare  and  Submit  the  TCP.  Now,  after  a  need  for  a  change 
is  identified  and  classified,  DOD-STD-480A  states  that  the  engineering 
change  proposal  will  be  prepared  for  submittal  to  the  government  (6:3). 
This  will  be  done  for  Class  II  engineering  changes,  which  will  then  be 
submitted  to  DCAS  for  concurrence  on  the  classification  (17).  However, 
for  Class  I  changes,  the  ACM  SPO  requires  a  Technical  Change  Package 
(TCP)  be  reviewed  by  the  SPO  before  the  ECP  is  prepared  (14).  Indeed, 
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A  TCM  between  the  Air  Force  and  the  contractor (s)  will  be  held 
during  the  preparation  cycle  of  all  formal  ECPs ,  and  as  required 
for  CCPs .  Likewise,  a  TCM  will  be  held  during  the  preparation 
cycle  of  the  formal  proposal  in  response  to  a  SPO  approved 
urgent/emergency  change  proposal.  The  TCM  will  be  held  after  the 
contractor  has  completed  and  made  available  the  descriptive 
portion  of  the  change  proposal  as  contained  in  the  TCP  and  before 
the  contractor's  pricing  of  that  proposal....  To  be  specific,  the 
only  exceptions  to  the  TCP/TCM  requirement  are  non- formal  urgent, 
emergency,  and  administrative  change  proposals  and  special  cases 
approved  by  the  program  manager  and  the  Director  of  Configuration 
Management.  (7:7) 

The  government  will  review  the  TCP  and  provide  comments  about  it  to  the 

contractor  at  the  Technical  Coordination  Meeting  (TCM) ;  these  comments 

will  assist  the  contractor  in  its  preparation  of  the  ECP  (7:7).  The  TCP 

includes  the  technical  details  of  the  change;  however,  it  does  not 

include  any  specific  or  firm  cost  data  because 

The  function  of  the  TCM  is  to  establish  the  baseline  as  defined  by 
the  documentation  wording  [TCP]  to  ensure  accuracy  and  adequacy 
before  the  contractor  expends  effort  in  pricing  the  proposal  for 
formal  submittal  to  the  SPO.  (7:7) 

Furthermore,  the  TCP  should  be  prepared  in  accordance  with  the 

requirements  outlined  in  DOD-STD-480A  for  preparing  ECPs  because  the 

TCP,  if  approved  as  written,  will  be  submitted  to  the  government  as  the 

technical  portion  of  the  ECP- -the  only  differences  being  the  cover 

letter  and  the  addition  of  a  cost  volume  (5) .  After  the  TCP  is 

prepared,  the  contractor  mails  copies  of  the  document  to  the  ACM  SPO, 

OCALC,  DCAS ,  and  MDMSC  (17).  Once  the  TCP  is  received  at  the  SPO,  three 

steps  take  place  which  lead  up  to  the  TCM.  These  three  steps  are 

Selecting  the  Office  of  Primary  Responsibility  (OPR) ,  Distributing 

Copies,  and  Reviewing  the  TCP. 

Select  the  OPR.  When  the  ACM  SPO  receives  the  ECP, 
the  organization  that  actually  receives  and  opens  the  document  (or  box, 
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for  voluminous  copies  of  ECPs)  is  VCC  (17).  VCC  then  determines  the 
appropriate  directorate  within  which  the  ECP  should  be  assigned  (17). 

The  directorate  is  determined  by  a  simple  test:  Is  the  ECP  so  specific 
that  it  is  a  concern  strictly  for  only  one  directorate  (7:Atch  1)?  If 
the  answer  is  yes,  then  the  OPR  will  be  chosen  from  within  that 
directorate  (7:Atch  1).  If  the  answer  is  no,  then  the  OPR  will  be 
chosen  from  within  the  Projects  Directorate,  VCA  (7:Atch  1).  VCC  will 
not  choose  the  specific  individual  within  that  directorate,  but  instead 
ask  the  directorate  to  make  the  selection  (7:Atch  1).  Usually,  roughly 
9  times  out  of  10,  the  OPR  is  chosen  from  VCA  (17). 

Distribute  Copies  of  ECP  within  the  SPO.  Once  VCC 
knows  who  the  OPR  is,  they  attach  a  cover  letter  to  the  front  of  the  ECP 
and  distribute  a  copy  to  each  directorate  in  the  SPO  (17).  The  cover 
letter  indicates  the  OPR,  the  date  by  which  the  TCP  must  be  reviewed, 
and  the  projected  date  of  the  TCM  (17).  When  the  other  directorates 
receive  their  copy  of  the  TCP,  a  reviewer  is  selected  to  comment  on  the 
TCP  with  respect  to  his  functional  expertise  (17).  In  some  instances, 
proposals  are  complex  enough  to  demand  two  or  more  personnel  from  the 
same  directorate  to  review  the  document;  this  is  especially  true  for 
VCE,  ^or  example,  where  one  ECP  may  require  review  by  multiple 
engi  -?  'ing  disciplines  (17). 

Review  TCP.  Witl.ln  the  time  allotted  to  them,  these 
"chosen"  reviewers  should  study  the  proposed  change  and,  if  necessary, 
take  note  of  any  issues  that  cross  their  mind  as  they  complete  the 
review.  A  frequent  question  many  have  asked  is,  for  what  should  the 
reviewer  look?  The  answer  is,  simply  enough,  anything  that  looks  out  of 
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the  ordinary  or  that  causes  one  to  wonder  about  the  answers  to  who, 
what,  when,  why,  or  how,  especially  within  the  area  of  one's  functional 
expertise.  Indeed, 

Although  the  contractor  has  the  responsibility  to  define  all 
impacts  of  a  proposed  change,  each  SPO  Directorate  shall  ensure 
that  all  impacts  to  their  specific  areas  of  responsibility  have 
been  identified  and  adequately  defined.  (7:3) 

After  the  review  is  completed,  the  reviewer  should  attempt  to  resolve 

any  issues  that  still  remain  by  preparing  to  ask  the  contractor 

questions  at  the  TCM  (17). 

Conduct  TCM.  The  purpose  of  the  TCM  is  "to  facilitate  CCB 

action,  speed  the  processing  of  the  change,  and  provide  understanding  of 

change  requirements  between  the  contractor  and  the  government"  before 

the  ECP  is  formally  submitted  (7:2).  The  TCM  is  described  below: 

TCMs  will  normally  be  conducted  by  teleconference  with  the 
appropriate  SPO  Change  Manger,  OPR,  Program  Manager,  other  SPO 
functionals  as  required  and  contractor  representatives.  If  the 
complexity  of  the  change  proposal  warrants,  the  TCM  may  be 
organized  and  conducted  at  a  contractor  facility  or  at  the  SPO 
with  all  representatives  being  present.  The  actual  TCM  date  shall 
appear  on  a  TCM  schedule  distributed  by  VCC  and  shall  be  a  func¬ 
tion  of  the  availability  of  required  people.  The  TCP  must  be 
available  to  the  SPO  7  to  10  days  in  advance  of  the  actual  TCM 
date.  The  OPR  shall  also  ascertain  if  significant  changes  are 
anticipated  from  the  Rough  Order  of  Magnitude  (ROM)  price  [a 
preliminary  estimate  of  the  possible  cost  of  implementing  the 
proposed  design  change]  provided  by  the  contractor  on  the  ACSN. 

The  TCM  is  an  informal  meeting  and  results  in  informal  assurance 
to  the  contractor  that  the  descriptive  portion  of  a  change  propo¬ 
sal  as  presented  in  the  TCP  is  accurate  and  adequately  defines  the 
change  identified  by  the  ACSN.  Concurrence  with  the  descriptive 
portion  of  the  proposal  by  those  in  attendance  at  the  TCM  should 
reflect  the  opinion  of  the  SPO  community  at  large  as  attested  by 
signature  of  the  OPR  and  Change  Manager  on  the  TCM  minutes.  These 
signatures  shall  be  acquired  by  the  Change  Manager  within  five 
days  of  receipt  of  minutes  from  the  contractor.  A  copy  of  the 
signed  TCM  minutes  shall  be  provided  to  the  contractor  by  VCC 
immediately  upon  completion  of  all  required  actions.  Once  a 
change  proposal  has  successfully  passed  the  TCM  milestone,  the 
contractor  should  have  received  all  necessary  technical  direction 
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required  to  complete  preparation  of  the  change  proposal.  Never¬ 
theless,  approval  of  the  proposal  cannot  be  assured  until  the  CCB 
has  the  opportunity  to  review  the  formal  submission  of  the 
proposal.  (7:7-8) 

So,  if  the  TCP  is  adequately  reviewed,  then  the  TCM  should  provide  the 
contractor  the  information  it  needs  to  complete  the  ECP,  including  the 
cost  volume  (7:8).  However,  if  the  TCP  is  not  adequately  reviewed,  not 
only  might  there  be  problems  with  the  technical  portion  of  the  change, 
but  also  with  the  cost  portion  of  those  changes  that  increase  the  scope 
of  the  contract.  Thus  the  TCM  review  process  is  perhaps  the  most  impor¬ 
tant  activity  for  the  SPO,  MDMSC,  DCAS ,  and  OCALC  personnel  to  perform 
in  a  satisfactory  manner  because  if  a  TCP  is  poorly  reviewed,  then  the 
quality  of  the  ECP  could  be  diminished.  For  example,  if  a  technical 
issue  is  overlooked,  then  the  validity  of  the  cost  volume  could  be 
impacted.  Furthermore,  either  important  considerations  will  remain 
unnoticed  or  the  issue  that  could  have  been  identified,  worked,  and 
corrected  before  the  ECP  is  prepared  may  become  an  issue  during  the  ECP 
review  process  that  could  delay  the  presentation  of  the  ECP  at  the  CCB 
(17).  In  either  ca~e,  the  consequences  of  an  inadequate  review  are 
severe,  resulting  in  either  poor  decisions  on  the  part  of  the  government 
or  lost  time  while  issues  are  worked  out  in  the  final  stages  of  the 
process  rather  than  in  the  beginning.  As  stated  above,  the  contractor 
and  the  SPO  perform  steps  to  document  the  results  of  the  meeting.  These 
steps  include  the  preparation  of  the  TCM  Minutes  by  the  contractor, 
submittal  of  the  minutes  to  the  SPO,  and  Approval  of  the  Minutes  by  the 
SPO  (7:8).  If  the  SPO  does  not  approve  the  minutes,  then  the  SPO  might 
request  the  contractor  to  update  the  TCP  for  purposes  of  conducting  a 
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second  TCM  (17).  In  either  case,  the  contractor  may  prepare  the  ECP  an 
submit  it  to  the  government  after  the  TCM  minutes  are  approved  (17). 

Prepare  the  ECP.  Immediately  after  a  need  for  a  change  is 
identified  and  classified,  the  contractor  may  submit  an  ECP  directly  to 
the  government  without  the  benefit  of  the  TCP/TCM  process  if  the 
proposal  is  a  "non-formal,  urgent,  emergency  [or]  administrative  change 
proposal  [or  if  it  is  a]  special  case  approved  by  the  program  manager 
and  the  Director  of  Configuration  Management"  (7:7).  In  these 
instances,  the  contractor  is  still  responsible  for  preparing  the  ECP  in 
accordance  with  the  requirements  stipulated  in  DOD-STD-480A  (6:12). 
Furthermore,  even  though  a  TCM  may  have  been  conducted  and  no  comments 
were  provided,  the  contractor  still  must  submit  an  ECP  to  the  SPO  so 
that  the  proposal  may  be  formally  approved  at  the  CCB  (7:8). 

Submit  ECP  to  the  Government.  Once  again,  if  GD/C  wants  to 
implement  a  Class  II  change,  the  ECP  needs  to  be  submitted  to  DCAS  for 
concurrence  that  the  change  is  indeed  a  Class  II  change  (17).  If  DCAS 
concurs  that  it  is  a  Class  II  change,  then  no  other  action  on  the  part 
of  the  ACM  SPO  is  necessary  (17).  However,  if  the  proposal  is  for  a 
Class  I  change,  then  the  contractor  must  make  sure  the  ECP  gets  to  the 
ACM  SPO  for  the  review  and  approval  process  because  the  ACM  SPO  is  the 
only  government  organization  with  approval  authority  of  ACM  Class  I  ECPs 
(17).  Furthermore,  the  contractor  submits  copies  of  the  ECP  to  MDMSC, 
OCALC ,  ana  DCAS  who  provide  comments  to  the  SPO  to  help  it  make  its 
decision  to  approve  the  proposal  (17).  Since  the  SAC  liaison  officer  is 
on  the  internal  ACM  SPO  mailing  list,  all  affected  organizations  receive 
copies  of  and  can  begin  reviewing  each  ECP  at  about  the  same  time  (17). 
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Review.  Now  that  all  the  organizations  have  received  a  copy 
of  the  ECP,  they  can  begin  the  review  process.  This  review  process  in¬ 
cludes  the  task  of  ensuring  that  the  proper  personnel  in  each  organiza¬ 
tion  are  provided  a  copy  of  the  proposal.  Suffice  it  to  say  that  DCAS , 
OCALC,  and  MDMSC  have  internal  controls  to  ensure  the  proper  personnel 
receive  a  copy  of  the  ECP.  These  internal  controls  and  review  processes 
will  not  be  discussed  here  because,  since  it  is  at  the  SPO  that  the 
decision  will  be  made  whether  or  not  to  approve  the  ECP,  this  thesis 
will  concern  itself  with  only  the  review  process  as  it  exists  within  the 
ACM  SPO.  Besides,  part  of  the  ACM  SPO  review  process  includes  obtaining 
comments  from  these  other  organizations;  hence,  if  any  of  the  review 
processes  at  these  other  organizations  is  untimely,  then  that  fact  would 
become  evident  through  the  study  of  the  ACM  SPO's  process  and  steps 
could  then  be  taken  to  research  the  other  organizations'  internal  review 
processes.  So,  confining  our  research  efforts  for  now  within  the  ACM 
SPO,  the  review  process  consists  of  disseminating  copies  of  the  ECP  to 
each  of  the  directorates,  reviewing  the  document,  compiling  the 
comments,  resolving  the  issues,  and  preparing  for  presentation  to  the 
CCB.  These  five  steps  are  outlined  below. 

Distribute  Copies  of  ECPs  within  the  SPO.  Since  VCC 
already  knows  who  the  OPR  is,  they  ask  him  when,  within  the  next  two  to 
three  weeks,  he  will  be  available  to  present  the  ECP  to  the  CCB  (17). 
Once  the  CCB  date  is  determined,  VCC  attaches  a  cover  letter  to  the 
front  of  the  ECP  and  distributes  a  copy  to  each  directorate  in  the  SPO 
(17).  The  cover  letter  indicates  the  OPR,  the  date  by  which  the  ECP 
review  comments  must  be  returned  to  the  OPR,  and  the  projected  date  of 
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the  CCB  (17).  When  the  other  directorates  receive  their  copy  of  the 
ECP,  a  reviewer  is  selected  to  comment  on  the  ECP  with  respect  to  his 
functional  expertise  (17). 

Review  ECP.  The  reviewers  review  the  proposal, 
looking  for  the  additional  information,  if  any,  that  was  requested  by 
the  government  at  the  TCM,  and  they  take  note  of  any  issues  that  cross 
their  mind  as  they  complete  the  review.  After  the  review  is  completed, 
the  reviewer  should  attempt  to  resolve  any  issues  that  he  may  have  by 
following  the  steps  outlined  below  in  the  "Resolving  Issues"  paragraph. 
For  those  issues  that  remain  outstanding,  he  should  discuss  them  with 
the  OPR  and,  if  necessary,  formally  document  those  issues  to  the  OPR 
(17) .  This  review  process  is  as  important  an  activity  for  the  govern¬ 
ment  personnel  to  perform  in  a  satisfactory  manner  because,  if  an  ECP  is 
poorly  reviewed,  either  important  considerations  will  remain  unnoticed 
or  the  issue  that  could  have  been  identified,  worked,  and  corrected 
before  the  CCB  may  become  an  issue  at  the  CCB  (17).  In  either  case,  the 
consequences  of  an  inadequate  review  are  severe,  resulting  in  either 
poor  decisions  on  the  part  of  the  government  or  lost  productivity  due  to 
CCB  board  members  and  others  in  attendance  having  their  time  wasted 
while  issues  that  could  have  been  resolved  earlier  are  brought  up, 
discussed,  and  go  unresolved  until  the  proper  actions  can  be  taken  to 
clear  up  the  issue. 

Compile  the  Comments.  As  the  SPO  ECP  OPR  begins  to 
collect  the  other  directorates'  comments,  he  can  begin  reviewing, 
compiling,  and  categorizing  them.  By  reviewing  the  others'  comments,  he 
may  be  able  to  see  some  concerns  that  can  be  answered  either  by  himself 
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or  with  the  help  of  someone  he  knows.  Compiling  the  comments  will  help 
him  keep  tabs  of  those  who  have  not  yet  responded,  and  it  will  allow  him 
to  group  similar  concerns  within  categories,  such  as  administrative  con¬ 
cerns  like  dotting  the  "i's"  and  crossing  the  "t's",  or  more  substantive 
content  or  process  concerns.  When  he  is  through  compiling  all  of  the 
comments,  including  those  from  MDMSC  and  OCALC,  he  is  ready  to  attempt 
to  resolve  any  outstanding  issues . 

Resolve  Issues.  Before  a  reviewer  provides  a 
response  to  the  OPR,  he  should  try  to  resolve  any  issues  he  has  with  the 
proposal.  First,  he  should  discuss  the  concern  with  fellow  SPO 
personnel,  maybe  even  the  OPR- -perhaps  someone  else  in  the  SPO  has  an 
answer  to  the  question.  Second,  the  government  personnel  at  DCAS  often 
have  insight  into  the  particulars  of  a  proposed  change.  Recall  that 
DCAS  personnel  attend  many  of  the  GD/C  meetings,  including  those  where 
future  ECP  submittals  are  discussed  in  detail,  so  consultation  with  them 
about  the  issue  can  prove  quite  productive.  After  exhausting  government 
resources,  if  the  issue  is  still  not  resolved,  a  call  to  the  contractor 
is  in  order.  Finally,  if  the  discussion  with  the  contractor  is  unpro¬ 
ductive,  the  issue  should  be  written  >  a  as  an  unresolved  or  unworkable 
issue  and  passed  on  to  the  ECP's  OPR  who  will  compile  the  comments  from 
all  of  the  other  reviewers.  The  OPR  should  also  attempt  to  resolve  any 
outstanding  concerns,  to  the  best  of  his  ability,  before  presenting  the 
proposal  to  the  CCB.  Perhaps  one  of  the  most  humbling  experiences  an 
OPR  can  experience  is  to  be  told  that  he  has  not  properly  worked  all  of 
the  issues.  Obviously,  all  issues  cannot  be  worked  by  the  directorate 
reviewers  or  the  OPR;  however,  an  attempt  must  be  made.  F  .hose 
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issues  which  remain  outstanding  even  until  the  CCB,  the  decision  to 
escalate  the  concern  above  the  level  to  which  it  has  been  worked  can  be 
made  by  the  CCB  chairman  (17). 

Prepare  ECP  for  CCB.  The  final  step  within  this 
overall  review  process  is  to  prepare  the  ECP  for  presentation  to  the 
CCB.  The  OPR  should  prepare  briefing  charts  which  will  concisely 
describe  the  change;  discuss  the  ramifications  to  cost,  schedule, 
logistics  supportability ,  and  performance  of  the  system  whether  or  not 
the  engineering  change  is  implemented;  the  effectivity  of  the  change 
(i.e.,  for  which  missiles  will  the  change  be  implemented,  and  for  which 
will  it  either  not  be  implemented  or  be  implemented  at  a  later  date  via 
a  retrofit);  all  outstanding  issues  and  concerns;  and  the  OPR's 
recommendation  (17).  The  OPR  must  also  indicate  whether  or  not  the 
change  will  impact  MDMSC's  efforts  to  produce  "build  to  print"  ACMs 
(17).  Once  the  charts  are  prepared,  the  OPR  is  ready  to  present  the  ECP 
to  the  CCB  for  a  disposition. 

Disposition  the  ECP.  The  CCB  has  been  mentioned  at  great 
length,  and  it  was  briefly  described  in  Chapter  I.  However,  a  more 
detailed  look  at  the  CCB  is  now  in  order  because  the  CCB  determines  the 
disposition  of  the  ECP.  In  light  of  this,  the  approval/disapproval 
process  will  be  discussed  in  conjunction  with  the  CCB. 

CCB  Authority.  In  accordance  with  VC  01  800-1,  the 

CCB  has  the  authority  to  disposition  ACM  ECPs . 

The  ACM  CCB  is  an  official  joint  command/agency  group  responsible 
tor  the  control  of  all  changes  proposed  to  the  scope  of  the  con¬ 
tract.  The  System  Program  Director  or  designated  alternate,  as 
chairperson,  is  solely  responsible  for  the  approval  or  disapproval 
of  a  proposed  change.  (7:8) 
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CCB  Membership.  Membership  on  the  CCB  is  established 


in  VC  01  800-1,  which  states  that 

CCB  membership  shall  be  limited  to  one  primary  member  and  one,  or 
more,  alternate  member(s)  from  each  directorate.  Members  and 
alternates  shall  be  assigned  by  and  identified  on  Special  Orders. 
(7:8). 

The  01  goes  on  to  state  that  membership  will  automatically  consist  of 
the  System  Program  Director  and  the  director  from  each  directorate  as 
the  chairperson  and  primary  board  members,  respectively,  and  that  the 
director  of  VCC  will  fulfil  the  special  role  as  CCB  Secretariat  (7:8). 

General  CCB  Decision  Rules.  The  CCB  has  been  provided 
general  guidelines  by  which  it  will  normally  base  its  decision: 

(1)  Change  proposals  intended  to  incorporate  improvements  or 
increase  system  capability  required  to  satisfy  an  operational 
requirement,  contractual  performance  or  design,  or  to  correct  a 
deficiency  shall  normally  be  approved. 

(2)  Change  proposals  to  incorporate  improvements  not  required  to 
meet  present  contract  requirements  or  to  increase  capability 
beyond  Program  Management  Directive  (PMD)  authorization  shall 
normally  be  disapproved.  (7:8) 

CCB  Procedures .  VC  01  800-1  also  provides  general  and 
specific  CCB  procedures.  Namely,  a  CCB  is  usually  held  once  each  week 
at  a  set  time;  however,  for  ECPs  needing  immediate  attention,  the  CCB 
chairperson  can  call  a  CCB  meeting  at  any  time  (7:8-9).  The  01  con¬ 
tinues  : 


It  is  important  to  recognize  that  the  following  procedures  iden¬ 
tify  the  steps  to  be  taken  to  ensure  thorough  evaluation  of  change 
proposals  received.  If  expedited  processing  is  required  for  a 
particular  change  proposal  these  steps  must  still  be  taken,  the 
only  difference  is  the  amount  of  time  permitted  for  total  process¬ 
ing.  the  following  procedures  will  be  followed  during  the  conduct 
of  the  CCB: 

(1)  Each  member,  or  their  alternate,  is  responsible  for 
providing  to  the  CCB  a  Directorate  recommendation  on  each  change 
proposal  to  be  reviewed.  All  comments/questions  concerned  with  a 
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change  proposal  shall  be  provided  to  the  OPR  at  least  three  days 
prior  to  the  scheduled  CCB .  A  copy  of  all  comments  shall  be 
forwarded  to  VCC  for  inclusion  in  the  official  file. 

(2)  CCB  attendance  shall  be  documented  by  the  signature  of 
the  CCBD  [Configuration  Control  Board  Directive,  the  form  that 
indicates  to  the  contractor  whether  or  not  the  CCB  was  approved, 
plus  any  comments  if  appropriate].  If  a  directorate's  primary  and 
alternate  members  cannot  attend  the  CCB,  all  CCBDs  for  that  meet¬ 
ing  will  be  annotated  with  the  word  "ABSENT"  in  the  appropriate 
signature  block. 

(3)  the  CCB  chairperson  shall  document  the  decision  for 
each  change  proposal  by  signing  all  applicable  sections  of  the 
CCBD.  For  ACSNs ,  the  chairperson  will  personally  sign  the  ACSN 
prior  to  submittal  to  the  contractor.  The  CCBD  shall  reflect  one 
of  the  decisions  below: 

(a)  Approve  the  proposal  as  written. 

(b)  Approve  the  proposal  contingent  upon  the  incor¬ 
poration  of  specific  changes  and  submittal  of  either  a  change  or 
revision  to  the  change  proposal  as  identified  on  the  CCBD. 

(c)  Disapprove  the  change  proposal.  The  reason  for 
disapproval  shall  be  stated  on  the  CCBD. 

(d)  Defer  the  proposal  for  further  investigation. 

Once  signed,  the  CCBD  shall  not  be  changed  without  approval  of  an 
amenoed  CCBD  by  the  chairperson. 

Each  member  shall  signify  their  concurrence  with  the  decision  by 
signing  the  CCBD  at  the  completion  of  the  CCB  unless  the  member 
"non-concurs"  with  the  chairperson's  decision.  In  such  cases,  the 
dissenting  member  shall  state  their  non-concurrence  during  the  CCB 
and  shall  submit  a  letter,  outlining  the  reason  for  non-concur¬ 
rence,  to  the  chairperson  (a  copy  must  be  forwarded  to  VCC  for 
inclusion  in  the  official  file)  within  three  working  days  after 
the  CCB  meeting.  Each  non-concurrence  shall  be  evaluated  and  a 
final  decision  shall  be  made  by  the  SPO  director.  (7:9) 

Incorporate  the  Approved  Chanee .  It  is  apparent  that  the 
CCB  has  specific  procedures  by  which  it  is  to  disposition  ECPs .  Speci¬ 
fic  procedures  also  exist  for  what  is  to  be  done  after  a  CCB  decision  is 
made.  Basically,  five  steps  must  be  performed  after  a  CCB  decision: 
notify  the  contractor  of  the  decision,  follow  up  on  conditions  of 
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approval,  participate  in  fact-finding  and  negotiating,  ensure  contrac¬ 
tual  agreements  are  finalized,  and  ensure  the  contractor  takes  the  steps 
to  implement  the  design  change  and  update  the  required  documentation. 

Publicize  the  Decision.  After  the  CCB,  one  of  the 
first  things  to  be  done  is  to  notify  GD/C,  MDMSC,  DCAS ,  and  OCALC  of  the 
CCB's  approval  decision  regarding  the  ECP  (17).  This  notification 
process  is  performed  by  preparing  and  completing  the  CCBD  and  sending  it 
to  these  organizations  (7:4).  For  an  ECP  that  has  been  approved  as 
written,  the  CCBD  is  the  only  document  that  needs  to  be  sent  out; 
however,  if  the  disposition  was  anything  but  approval,  then  a  document 
needs  to  be  sent  along  with  the  CCBD  that  explains  the  CCB's  decision  in 
further  detail  (17).  For  example,  if  the  ECP  is  dispositioned  as 
"Approved  with  Comments,"  then  a  document  needs  to  be  attached  to  the 
CCBD  that  explains  what  the  comments  are. 

Follow  Up  on  CCBD  Conditions.  The  OPR  should  expect  a 
response  from  the  contractor  whenever  comments  are  sent  out  with  the 
CCBD.  For  an  ECP  that  was  Approved  with  Comments,  the  contractor  would 
respond  by  submitting  an  updated  copy  of  the  ECP  that  incorporates  all 
of  the  changes  that  were  requested  by  the  CCBD  (17)  .  If  this  updated 
copy  includes  all  of  the  changes  directed  by  the  CCB,  then  the  OPR  and 
VCC  may  approve  the  changed  submittal  and  the  next  process  step,  "fact 
finding"  and  "negotiation,"  may  commence.  However,  a  different  response 
could  be  a  contractual  letter  explaining  why  the  contractor  will  not  be 
able  to  comply  with  one  or  more  of  the  comments,  in  which  case  the  CCB 
can  decide  to  either  back  off  from  its  original  requirement  or  hold  firm 
to  it  (17).  In  any  event,  if  the  government  and  the  contractor  do  not 
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agree  on  a  particular  CCBD  condition,  then  the  remaining  steps  of  the 
incorporation  process  may  get  delayed  until  agreement  is  reached.  Once 
the  government  approves  the  ECP  as  presented  by  the  contractor,  fact 
finding  and  negotiation,  if  necessary,  may  begin. 

Fact  Find  and  Negotiate  (Contractual  Review).  The 
cost  of  implementing  engineering  changes  that  are  required  to  make  the 
hardware  perform  up  to  contractual  specifications  do  not  get  added  to 
the  contract  because  the  costs  to  produce  hardware  to  that  specifica¬ 
tion  have  already  been  negotiated  into  the  original  contract.  However, 
for  those  ECPs  which  make  improvements  to  hardware  over  and  above  what 
the  contract  previously  stipulated,  the  contract  base  price  may  need  to 
be  adjusted.  But  before  the  contract  price  gets  adjusted,  the  costs 
outlined  in  the  ECP  need  to  be  studied  by  the  government  and,  when 
necessary,  defended  by  the  contractor.  These  processes  are  called  "fact 
finding"  and  "negotiation."  Fact  finding  is  an  attempt  by  the  govern¬ 
ment  to  ensure  that  the  costs  outlined  in  the  ECP  are  reasonable  (17). 
Costs  that  are  identified  as  unreasonable  are  brought  up  to  the  contrac¬ 
tor,  who  is  allowed  to  attempt  to  justify  those  costs  (17).  A  good, 
sound  fact  finding  effort  can  possibly  save  the  government  money,  but 
the  converse  if  true  also.  When  the  fact  finding  effort  is  completed, 
the  final  price  can  then  be  negotiated.  For  ease  of  discussion,  these 
two  steps  will  be  referred  to  simply  as  "contractual  review"  through  the 
remainder  of  this  report. 

Finalize  Contract  Modification.  After  a  negotiated 
price  is  reached,  the  final  step,  contract  modification,  can  be  accom¬ 
plished.  After  the  ECP  is  approved  at  the  CCB  and  any  C  -rtf)  issues  have 
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been  resolved,  the  OPR  and  a  person  from  VCK  work  together  to  complete 
the  fact  finding,  negotiation,  and  contract  modification  efforts.  The 
OPR  is  still  responsible  for  completion  of  all  these  steps;  however,  the 
OPR  cannot  adequately  perform  these  jobs  without  assistance  and  involve¬ 
ment  from  the  contracts  representative,  especially  for  this  final  ECP 
processing  step.  When  the  change  is  put  on  contract,  the  proposal 
becomes  a  contractually  required  task  that  the  contractor  must  perform 
if  it  wishes  to  stay  within  the  constraints  of  the  contract. 

Ensure  Actual  Implementation.  Usually,  the  contractor 
does  not  need  any  encouragement  to  implement  the  approved  change 
because,  if  the  design  change  is  one  that  corrects  a  design  deficiency, 
the  contractor  knows  that  the  government  will  not  accept  hardware 
without  the  incorporated  change  (17).  However,  there  are  times  when  the 
contractor  may  feel  it  is  to  his  advantage  to  delay  implementation  of  a 
change,  in  which  case  the  OPR  must  monitor  the  contractor's  progress  and 
"encourage"  him  to  make  progress  (17).  Usually,  only  after  the  change 
is  fully  implemented  is  the  OPR's  responsibility  with  respect  to  that 
particular  ECP  relieved  (17) . 

ECP  and  ECP  Process  Summary.  This  section  has  described  the  ACM 
SPO  ECP  process  and  it  has  described  all  of  the  steps  in  the  process, 
from  determining  a  need  for  a  change  to  actual  implementation  of  the 
change.  Since  the  first  hypothesis  asserts  that  the  ECP  process  is 
untimely,  the  reader  needs  to  know  how  much  time  the  ACM  SPO  believes  it 
should  take  to  implement  an  ECP  through  its  process.  The  next  section 
outlines  these  time  goals. 
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The  Time  Requirements 

Although  DOD-STD-480A  provides  no  firm  time  requirement  within 
which  routine  ECPs  must  be  processed,  the  general  guideline  which  the 
ACM  SPO  has  historically  followed  is  to  take  roughly  180  days  to  put  a 

low-cost  proposal  on  contract  after  the  TCP  is  first  received  by  the  SPO 

(14).  Within  this  general  time  frame,  the  ACM  SPO  and  GD/C  jointly 
developed  a  schedule  and  made  the  schedule  a  part  of  the  contract 

between  the  two  organizations  (14) .  The  schedule  is  included  in  ACM- 

DOC-301,  the  ACM  Configuration  Management  Plan  (CMP),  which  stipulates 
which  organization  is  responsible  for  completing  which  steps  of  the 
process,  from  identification  of  the  need  for  a  change  to  the  start  up  of 
the  implementation  process,  and  within  how  much  time  (14).  These  steps 
are  described  below  and  the  amount  of  time  allowed  to  complete  them  is 
shown  in  Table  2. 


TABLE  2 

REQUIRED  TIMES  FOR  PROCESS  STEPS 


Organization 

Step  From 

/  To 

Number  of  Davs 

GD/C 

/  Submit  TCP 

Begin 

ACM  SPO 

TCP 

/  Conduct  TCM 

14 

ACM  SPO 

TCM 

/  Submit  ECP 

38 

ACM  SPO 

ECP 

/  Conduct  CCB 

28 

ACM  SPO 

CCB 

/  Submit  CCB  Comments  7 

GD/C 

Comments 

/  Submit  Response 

10 

ACM  SPO 

and  GD/C 

Response 

/  Complete  Review 

40 

ACM  SPO 

and  GD/C 

Review 

/  On  Contract 

10 

Total  for  no-cost  proposals 

147 

Total  for  proposals 

from  $0  to 

$0.5  million 

217 

Total  for  proposals 

from  $0.5 

to  $3.5  million 

237 

Total  for  proposals 

from  $3.5 

to  $25  million 

257 
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TCP  to  TCM.  The  ACM  SPO  has  14  days  to  review  the  TCP  and  conduct 
the  TCM  after  it  receives  the  TCP  (9:3-2). 

TCM  to  ECP.  Within  five  days  after  the  TCM,  the  contractor  must 
submit  TCM  minutes  which  annotate  the  results  of  the  TCM  (9:3-2).  The 
SPO  then  must  approve  those  minutes  as  accurate  (9:3-2).  The  contractor 
then  has  33  days  to  submit  the  ECP  to  the  SPO  (9:3-2).  If  the  SPO  does 
not  approve  the  minutes,  then  the  contractor  and  the  SPO  must  meet  again 
to  determine  the  proper  course  of  action. 

ECP  to  CCB.  The  SPO  has  28  days  to  review  the  ECP  and  conduct  the 
CCB  (9: 3-2a) . 

CCB  to  Complete  Review.  After  the  CCB,  the  two  organizations  have 
57  days  to  complete  the  contractual  review  of  a  no-cost  change  (9:3- 
2a).  During  this  time,  the  SPO  has  seven  days  to  submit  its  CCBD  and 
the  contractor  has  10  days  to  respond  to  the  CCBD  (9 : 3-2 , 3-2a) .  This 
response  could  either  be  a  letter  signifying  acknowledgement  of  the 
CCB's  decision  to  approve,  defer,  or  disapprove  the  ECP,  or  it  could  be 
an  updated  or  revised  submittal  of  the  ECP  based  on  the  CCB's  decision 
to  approve  the  ECP  with  comments  (17).  Then,  for  no-cost  ECPs ,  GD/C  has 
30  days  to  submit  a  draft  contract  modification  (mod)  and  the  SPO  has  10 
days  to  review,  comment  on,  and  return  the  draft  mod  (9:3-2). 

Complete  Review  to  S/A.  For  no-cost  ECPs,  GD/C  and  the  SPO  have 
10  days  to  formally  sign  the  contract  mod  (9:3-2). 

Total  Time.  Summing  these  individual  step  times  produces  the 
overall  total  time  to  complete  the  processing  of  a  no-cost  ECP:  147 
days.  The  CMP  allows  more  time  to  complete  the  proposal's  processing  as 
the  cost  of  the  proposed  change  goes  up  Namely,  as  the  costs  rise,  the 
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CMP  provides  more  time  for  the  contractual  review  steps  (9:3-2).  The 
CMP  also  allows  extra  time  to  put  the  proposal  on  contract  after  the 
review  is  completed  (9:3-2).  For  a  proposal  costing  from  zero  to  one- 
half  million  dollars,  25  more  days  are  allowed  to  complete  fact  finding, 
20  more  days  to  complete  pre-negotiation  efforts,  and  25  more  days  to 
complete  the  negotiation  process,  totaling  of  70  days  (9:3-2).  For 
proposals  between  $500,000  and  $3.5  million,  15  extra  days  are  allowed 
for  the  three  additional  review  process  steps  plus  five  extra  days  are 
allowed  to  put  the  mod  on  contract,  for  a  total  of  20  extra  days;  and 
for  proposals  between  $3.5  million  and  $25  million,  15  extra  days  are 
allowed  for  the  review  process  and  five  extra  days  are  allowed  to  put 
the  mod  on  contract,  for  20  extra  days  total  (9:3-2).  It  is  unclear 
what  the  specific  ramifications  are  for  either  organization  if  one  or 
the  other  fails  to  meet  the  time  constraints  established  in  the  CMP, 
since  no  "penalties"  are  enforced  when  either  organization  fails  to  meet 
its  time  requirement  (14).  However,  if  delays  exist,  regardless  of  who 
causes  the  delay,  the  overall  impact  to  the  United  States  may  be  a  delay 
in  obta  ..ing  the  needed  level  of  deterrence. 

Efforts  to  Improve  Timeliness . 

As  the  reader  can  imagine,  based  on  the  discussion  above,  the 
engineering  change  implementation  process  is  quite  complex  and  involves 
many  people,  especially  when  one  considers  that  there  are  perhaps  90  or 
more  outstanding  ECPs  at  any  one  time  in  the  ACM  SPO  that  are  in  their 
various  stages  of  the  implementation  process  (17).  This  section  addres¬ 
ses  the  researcher's  and  the  ACM  SPO's  past  efforts  to  improve  the 
timeliness  of  the  specific  ACM  SPC  ECP  process  and  the  results  of  a 
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literature  search  for  information  regarding  the  efforts  of  other 
organizations  to  improve  their  internal  ECP  processes. 

The  Researcher's  Efforts  to  Improve  Timeliness.  Because  of  the 
researcher's  assignment  in  the  ACM  SPO,  he  became  aware  of  the  ECP 
process  and  the  need  to  study  it.  This  section  addresses  both  the 
researcher's  prior  assignment  in  the  ACM  SPO  and  his  previous  attempts 
to  improve  the  process. 

Researcher's  Assignment  in  the  ACM  SPO.  This  thesis  topic 
did  not  come  to  the  researcher  "out  of  the  blue."  While  a  project 
manager  in  the  ACM  SPO  from  14  November  1986  through  25  May  1989,  the 
researcher  was  the  ACM  Retrofit  Manager.  A  "retrofit"  is  a  process 
which  involves  a  change  in  configuration  of  a  fielded  weapon  system 
(17).  This  change  could  be  in  the  form  of  a  removal  and  replacement 
with  a  new  or  different  piece  of  hardware  or  software;  removal, 
effecting  the  engineering  change  on  the  part  that  was  removed,  and 
reinstallation  of  the  changed  piece  of  hardware  or  software;  or  instal¬ 
lation  of  a  new  piece  of  hardware  or  software  (17). 

Problem  Recognition.  Retrofits,  by  definition,  imply 
change;  those  changes  are  approved  through  the  ECP  process  (6:41-42). 
Furthermore,  the  ultimate  configuration  of  the  ACM  was  not  yet  stabil¬ 
ized,  while  the  researcher  was  the  retrofit  manager,  even  though  ACMs 
had  been  built.  This  instability  was  a  result  of  concurrent  design  and 
production  efforts  which  meant  that  production  of  hardware  was  imple¬ 
mented  before  all  of  the  design  work  was  finalized.  So,  because  the 
researcher  was  the  Retrofit  Manager  during  a  time  of  a  changing 
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configuration,  the  researcher  became  quite  intimate  with  the  symptoms  of 
the  problems  with  the  ECP  process. 

Attempts  to  Resolve  Problems.  Intimacy  with  the  symptoms  of 
the  process  problems  bred  a  desire  to  fix  the  problems.  So  the 
researcher  held  meetings  with  various  groups  inside  and  outside  the  SPO 
attempting  to  work  solutions  to  the  untimeliness  problems.  The 
researcher  discussed  the  issue  with  government  personnel  within  the  SPO 
and  at  OCALC  and  with  contractor  personnel,  all  of  whom  had  a  vested 
interest  in  resolving  the  untimeliness  problem.  Recommended  solutions 
were  proposed  and  implemented;  however,  the  problems  persisted.  These 
proposed  solutions,  and  the  impact  they  had  on  the  process  once  they 
were  implemented,  are  presented  next. 

Specific  Recommendations.  Prompted  by  the  researcher's 
perception  that  the  ECP  process  could  be  more  timely,  the  researcher 
made  a  recommendation  to  the  CCB  Chairman  to  begin  using  both  a 
Technical  Coordination  Meeting  (TCM)  and  a  control  to  lower  the 
incidence  of  OPR- induced  delays  in  meeting  the  CCB. 

Technical  Coordination  MeetinE.  Because  the  contrac¬ 
tor  was  having  difficulty  submitting  clearly  defined  and  written  propo¬ 
sals,  the  researcher  felt  that  a  meeting  between  the  government  and  the 
contractor,  before  the  ECP  was  submitted,  could  help  solve  the  problem 
with  the  poorly  written  proposals.  At  this  meeting,  the  organizations 
involved  with  the  ECP  process  would  review  an  informal  copy  of  the  yet- 
to-be-submitted  ECP,  called  the  Technical  Change  Package  (TCP).  The  TCP 
w<->uld  be  presented  as  the  technical  portion  of  the  ECP,  including  all  of 
the  technical  details  but  none  of  the  costs  (7:7).  The  purpose  of  the 
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TCM  was  "to  facilitate  CCB  action,  speed  the  processing  of  the  change, 
and  provide  understanding  of  change  requirements  between  the  contractor 
and  the  government"  before  the  ECP  was  formally  submitted  (7:2). 

Specific  steps  were  added  into  the  SPO  ECP  process,  but  it  was 
anticipated  that  these  steps  would  diminish  the  total  amount  of  time 

required  to  process  ECPs ,  increase  the  quality  of  the  result  of  the  ECP 

process,  or  both.  Recall  from  Table  2  above  that  the  CMP  allows  the  ACM 

SPO  14  days  to  review  the  TCP  and  another  38  days  are  allowed  before  the 

ACM  SPO  should  receive  the  ECP.  However,  it  was  believed  that  these  52 
extra  days  would  possibly  reduce  the  totai  processing  time  in  the  SPO, 
especially  if  the  TCP  review  and  TCM  performed  as  they  were  intended. 

Control  Delays  to  CCB.  A  minor  problem  that  existed 
in  mid  1988  was  the  cancelling  of  appearances  before  the  CCB,  by  the 
OPR,  right  before  he  was  scheduled  to  present  an  ECP  for  approval.  The 
CCB  members  took  time  to  review  copies  of  those  ECPs  scheduled  for  the 
next  CCB  so  that  they  could  make  an  approval  recommendation  to  the  CCB 
chairman.  However,  when  OPRs  cancelled  their  appointments  with  the  CCB 
at  the  last  minute,  they  effectively  wasted  the  time  the  CCB  members  had 
devoted  to  be  prepared  to  make  their  recommendation  because,  at  the  next 
CCB,  the  CCB  members  would  have  to  take  time  to  refresh  their  memories 
with  the  details  of  the  ECP.  Since  no  controls  existed  to  curb  the 
frequency  of  this  event,  it  was  believed  that  a  minor  control  procedure 
could  be  implemented  to  help  the  situation.  Thus  a  policy  was  put  in 
place  to  require  an  OPR  to  obtain  approval  for  a  delay  in  a  scheduled 
CCB  appearance  from  the  CCB  chairman.  This  policy  was  implemented; 
however,  the  control  measure  seems  to  have  since  ceased  to  function  as 
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it  was  intended  (16).  Both  deputy  program  directors  recently  thought 
that  the  other  deputy  director  was  being  asked  for  and  approving  delays 
in  presentations;  however,  neither  one  was  (16).  The  impact  of  these 
changes  has  been  in  question,  as  explained  below. 

Studying  the  Impact  on  Timeliness.  One  problem  with 
these  efforts  is  that  the  recommendations  were  made  based  mainly  on 
perceptions  about  the  ECP  process,  not  on  concrete  data.  Thus,  there 
was  not  much  data  to  confirm  that  the  problem  was  diminished  because  of 
the  implementation  of  the  recommendations .  Perhaps  the  lack  of  a 
rigorous  study  into  the  problems  in  the  first  place  is  why  the  problems 
persisted.  At  any  rate,  the  purpose  of  the  following  efforts  in  the  ACM 
SPO  is  to  more  rigorously  study  the  process  and  develop  measurement 
tools  which  will  indicate  a  change  in  the  timeliness  of  the  process. 

ACM  SPO  Efforts.  The  ACM  SPO's  efforts  to  improve  the  timeliness 
of  the  ECP  process  can  be  discussed  in  two  phases. 

First  Phase.  Since  the  timeliness  and  workload  problems 
continued  to  exist,  the  ACM  SPO  formed  a  "critical  process  improvement 
team"  (CPIT)  in  October,  1989  (16).  This  team,  made  up  of  government 
and  contractor  personnel,  will  attempt  to  improve  the  efficiency  of  the 
ECP  process  by  first  attempting  to  "come  to  a  full  understanding"  of  the 
current  ECP  process  so  that  through  understanding  it  might  be  able  to 
implement  techniques  that  will  improve  the  efficiency  of  the  process 
(16).  The  CPIT  conducted  a  survey,  using  a  questionnaire,  within  the 
ACM  SPO  to  help  the  team  understand  the  problems  experienced  by  those 
individuals  who  were  actively  involved  in  the  process.  A  copy  of  the 
survey  is  included  in  Appendix  B,  and  partial  results  of  this  survey 
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will  be  included  in  Chapter  IV.  Furthermore,  VCC  has  been  involved  with 
these  efforts  by  being  active  members  of  the  CPIT  and  by  developing  an 
operating  instruction,  VC  01  800-1,  which  outlines  the  major  require¬ 
ments  of  all  the  ECP  process  players  in  the  SPO  and  provides  guidelines 
as  to  how  long  the  steps  in  the  process  should  take;  the  01  time  guide¬ 
lines  are  slightly  more  ambitious  than  the  times  stated  in  the  CMP  (5). 
Furthermore,  the  CPIT,  after  studying  the  ECP  process,  will  make  recom¬ 
mendations  that  may  improve  the  process'  timeliness  and  effectiveness. 

Second  Phase .  Finally,  the  second  phase  is  represented  by 
the  researcher's  new  efforts  at  quantifying  the  ECP  process  in  conjunc¬ 
tion  with  the  efforts  of  the  CPIT.  The  researcher  will  perform  the 
steps  outlined  in  Chapter  III  so  that  specific  recommendations  can  be 
made  to  improve  the  timeliness  of  the  ECP  process. 

Efforts  of  Others  to  Improve  Their  ECP  Process.  To  determine  what 
other  organizations  have  done  to  improve  the  timeliness  or  efficiency  of 
their  configuration  change  approval  process,  a  search  of  the  literature 
was  performed.  The  Defense  Technical  Information  Center  (DTIC)  searched 
government  technical  reports  for  the  key  words  "engineering/modifica¬ 
tion,"  and  DIALOG  searched  the  public  literature  for  the  key  words 
"production,  engineering,  and  (configuration  or  design)  changes."  This 
search  produced  nothing  dealing  with  efforts  to  improve  the  engineering 
design  change  process. 

Chapter  Summary 

Chapter  I  presented  specific  hypotheses  about  the  timeliness  of 
the  ECP  process  and  the  ability  of  the  SPO  to  effectively  perform  the 
ECP  process.  This  Chapter  provided  the  background  information  needed  to 
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fully  explain  the  hypotheses.  Included  in  this  chapter  were 
descriptions  of  the  ACM  SPO,  the  outside  agencies  the  ACM  SPO  works  with 
when  processing  ECPs ,  the  ECP  and  the  ECP  process  steps  and  time 
requirements  within  the  SPO,  and  the  researcher's  and  others'  prior 
efforts  to  improve  the  timeliness  of  the  ECP  process.  Chapter  III  now 
will  address  the  specific  steps  that  the  researcher  will  follow  to  test 
his  hypotheses.  The  results  of  these  tests  will  be  presented  in 
Chapter  IV  and  they  will  be  used  to  make  the  recommendations  that  will 
be  presented  in  Chapter  V. 
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III.  Methodology 


Now  that  the  background  information  has  been  presented,  this 
chapter  will  provide  the  methodology  that  will  be  used  to  test  the 
hypothesis  about  the  process  timeliness  and  the  four  hypotheses  that 
claimed  that  specific  problems  exist  with  the  ECP  process.  For  each 
hypothesis,  the  discussion  will  follow  the  same  general  flow.  Namely, 
focus  questions  (FQs)  will  be  developed  that  convert  the  hypothesis 
statements  into  yes  or  no  questions;  assumptions  will  be  stated,  the 
specific  methods  used  to  determine  the  answer  to  the  focus  question  will 
be  described,  and  the  null  and  alternate  hypotheses  will  be  provided. 

The  chapter  is  divided  into  two  sections.  The  first  section  develops 
the  techniques  for  measuring  the  process  timeliness,  and  the  second 
section  develops  the  methods  that  will  be  used  to  confirm  or  deny  the 
hypothesis  claims  about  the  process  problems. 

Timeliness  of  ECP  Process 

Recall  that  the  first  hypothesis  states  simply  that  the  ACM  SPO 
ECP  process  is  not  timely.  This  hypothesis  will  be  tested  using  five 
focus  questions  (FQs).  For  each  FQ,  the  assumptions,  methods,  and  null 
hypotheses  are  presented.  These  five  FQs  include  the  following.  1)  Is 
the  ACM  SPO  ECP  process  taking  too  much  time  to  complete?  2)  Are  ECP 
process  steps  taking  too  much  time?  This  question  will  be  addressed  to 
each  process  step  to  determine  which  ones  are  untimely.  3)  Are  ACM  SPO 
ECPs  being  routed  through  the  same  process  steps  more  than  one  time? 

4)  Are  there  any  specific  ECP  process  flow  paths  that  are  more  timely 
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than  the  tiroes  presented  in  the  CMP?  5)  Is  there  a  disproportionate 
number  of  open  ECPs  awaiting  processing  in  any  of  the  ECP  process  steps? 
Figure  8  below  shows  the  general  order  of  the  following  discussion.  If 
the  first  focus  question  is  answered  positively,  then  the  four  other 
focus  questions  will  be  answered  to  perhaps  identify  specific  process 
steps  as  problem  areas. 

FO  Number  1 .  Is  the  ACM  SPO  ECP  process  taking  too  much  time? 

This  FQ  is  the  major  question  in  this  study.  All  of  the  following  FQs 
are  based  on  the  assumption  that  the  answer  to  this  question  is  "yes." 

Assumptions  for  FQ  Number  1.  Six  assumptions  will  be  made 
to  answer  this  focus  question. 


FIRST,  ASK 


ECP  PROCESS  TIMELINESS 


/ 

_ 

FQ  «  1:  PROCESS  UNTIMELY? 


F  ANSWER  IS  YES,  THEN  ASK 


o  '0  STATED 
o  ASSUMPTIONS 
o  ME’HOD 
o  HYPC’HlSlS 
0  ANSWER  >C 


FQ  0  2;  PROCESS  STEPS  UNTIMELY? 

— 

1 - 

:  o  STATED 

o  EQ  STATED 

1  o  ASSUMPTIONS 

o  ASSUMPTIONS  | 

:  ,  o  Mt'HOD 

o  method 

o  HYPC’nrsrs 

c  HYPOTHESES  j 

tm  :  o  ANSWER  rC 

o  ANSWER  "Q 

_ I _ 

FO  »  3:  ECPs  RE-RCJTED  THR0J0H  STE=S? 


FO  A  5:  OPEN  ECPs  DISPROPORTIONATELY 
DISTRIBUTED? 


o  to  S'ATEj 


I 

T _ 

-j  FC  #  4:  A'JY  PATHS  TME.Y? 


o  ASSUVPTCNS  i 
o  METHOD 
o  HYPOTHESES  , 
o  ANSWER  rc 


RESULT:  INDICATION  OF  THE 
TIMELINESS  OF  THE  PROCESS 


c  ro  s'a'cd 
c  ASSJM-'ONS 
c  METHOD 
c  H’POtHfS-S 
c  ANSAE-  E3 


Figure  8.  Discussion  Flow  for  Determining  Process  Timeliness 
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One .  Table  3  below  restates,  according  to  the  CMP, 
how  much  time  ECPs  should  take  to  be  processed  from  TCP  receipt  to 
supplemental  agreement.  The  first  assumption  is  that  the  time  provided 
by  the  CMP  is  an  appropriate  measure  with  which  to  compare  the 
timeliness  of  the  ECP  process  in  the  ACM  SPO.  So,  if  the  actual  process 
times,  as  determined  using  the  methods  presented  below,  are  less  than  or 
equal  to  those  spelled  out  in  the  CMP,  then  the  process  will  not  be 
considered  untimely.  However,  if  the  actual  process  times  are  more  than 
the  CMP  times,  then  the  answer  to  the  FQ  will  be  "yes."  This  assumption 
essentially  ignores,  for  the  moment,  one  key  p-.-int:  the  CMP  time 
standard  could  be  too  aggressive.  In  other  words,  perhaps  the  ECPs  are 
not  being  processed  within  the  CMP  time  constraints  because  the  CMP  is 
too  optimistic  in  how  much  time  an  ECP  should  take  to  be  processed. 


TABLE  3 

PROCESS  STEP  TIMES  STIPULATED  IN  THE  CMP  (9:3-2) 


Number  of  Davs 

Total 

for  no-cost  proposals 

147 

Total 

for  proposals  from  $0  to 

$0.5  million 

217 

Total 

for  proposals  from  $0.5 

to  $3.5  million 

237 

Total 

for  proposals  from  $3.5 

to  $25  million 

257 

Two .  The  amount  of  time  allowed  in  the  CMP  for 
processing  no-cost  ECPs  will  be  used  as  the  standard  to  answer  this 
focus  question.  Thus  the  actual  processing  time  of  the  selected  ECPs 
will  be  compared  to  the  value  of  147  days  (9:3-2). 
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Three .  The  second  assumption  above  stipulates  that 


only  no-cost  ECPs  will  be  studied.  In  addition  to  this  limitation,  the 
universe  from  which  ECPs  will  be  chosen  will  be  narrowed  even  more  by 
selecting  only  those  ECPs  that  were  approved;  were  processed  with  the 
benefit  of  TCP  Receipt,  TCP  Review,  and  TCM  process  steps;  and  had  any 
activity,  from  TCP  receipt  to  supplemental  agreement,  during  the  seven 
month  period  from  June  1,  1989  to  December  31,  1989.  The  assumption 
here  is  that  this  sample  of  ECPs  will  provide  a  true  picture  of  the 
overall  ECP  process  timeliness  regardless  of  the  cost  of  the  ECP. 
Additionally,  the  data  collected  with  this  time  criteria  will  be  assumed 
to  be  representative  of  the  process  as  it  existed  not  only  in  the  last 
seven  month  period  of  1989,  but  also  through  July  1990. 

Four  and  Five.  The  data  will  be  collected  from  indi¬ 
vidual  ECP  reports  that  are  produced  by  a  computer  database  program. 
These  reports  are  updated  periodically  by  the  OPR  and  the  VCC  Change 
Manager.  The  fourth  assumption  is  that  the  selected  reports  constitute 
a  random  sample  that  is  representative  of  the  true  situation.  Further¬ 
more,  the  fifth  assumption  is  that  these  reports  are  accurate  and  up  to 
date;  inaccurate  and  dated  material  will  impact  the  results  of  the 
study . 

Six.  Statistics,  including  the  sample  mean  and  sample 
standard  deviation,  will  be  computed  using  the  set  of  the  number  of  days 
the  ECPs  took  to  be  processed.  Since  the  sample  size  will  be  greater 
than  30,  the  sixth  assumption  is  that  the  Central  Limit  Theorem  applies; 
thus  the  sample  mean  is  approximately  normally  distributed,  the  expected 
value  of  the  sample  mean  is  equal  to  the  population  mean,  and  the  sample 
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variance  is  equal  to  the  population  variance  divided  by  the  sample  size 
(11:213).  This  assumption  allows  the  researcher  to  use  the  z  distri¬ 
bution  to  compute  the  90%  one-sided  confidence  interval  described  below. 

Method  for  FQ  Number  1.  The  FQ  will  be  answered  by 
comparing  the  actual  times  to  complete  the  processing  of  ECPs  to  the 
amount  of  time  stated  in  the  CMP. 

Collect  the  Data.  The  data  will  be  collected  from 
ECPs  that  never  were  considered  as  cost  ECPs,  that  first  arrived  in  the 
SPO  as  a  TCP,  that  completed  all  steps  necessary  to  be  put  on  contract, 
and  that  had  any  activity,  from  TCP  receipt  to  supplemental  agreement, 
during  the  last  seven  months  of  1989.  The  data  that  will  be  collected 
are,  for  each  individual  ECP,  the  ECP  number  and  the  actual  dates  of  two 
milestone  events.  These  two  milestone  completion  dates  are  the  date 
when  the  ACM  SPO  received  the  TCP  and  the  date  the  ECP  was  made  a  part 
of  the  contract.  These  two  dates  will  be  will  be  used  to  compute  the 
time  it  takes  the  ACM  SPO  to  process  ECPs  from  TCP  receipt  to  S/A. 

Manipulate  the  Data.  For  each  ECP,  the  two  milestone 
dates  will  be  put  into  a  QUATTRO  PRO  spreadsheet.  The  "(3DATEVALUE- 
(string)"  function  will  be  used  to  convert  the  dates  into  numerical 
equivalents.  For  example,  by  entering  the  dates  as  text  strings  in  the 
format  of  Month/Day/Year ,  the  @DATEVALUE( string)  function  can  be  used  to 
determine  the  number  of  days,  by  taking  the  difference  between  the  two 
numerals,  that  the  SPO  took  to  put  the  change  on  contract  after  the  SPO 
received  the  TCP  (4:30).  The  result  will  be  a  table  consisting  of  a 
column  of  numbers  in  the  spreadsheet  that  represent  the  number  of  days 
the  ECPs  took  to  be  processed. 
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Determine  the  Sample  Statistics.  Next,  QUATTRO  PRO'S 
@AVG  and  @STDS  functions  will  be  used  to  determine  the  sample  mean  and 
sample  standard  deviation  of  the  data.  These  sample  statistics  will  be 
used  to  calculate  a  90%  one-sided  confidence  interval  to  the  numerically 
lower  side  of  the  sample  mean.  The  lower  bound  value  (LBV)  of  this 
interval  will  be  computed  using  Equation  1: 

LBV  =  MT  -  Za  *  (S/Jn)  (1) 

where  MT  is  the  sample  mean  time,  Za  equals  1,28?  based  on  a  90%  one¬ 
sided  confidence  interval,  S  is  the  sample  standard  deviation,  and  n  is 
the  number  in  the  sample  (11:634).  The  significance  of  the  lower  bound 
value  is  that  there  is  a  90%  probability  that  the  population  mean  is 
above  it,  and  a  10%  chance  that  the  population  mean  is  below  it. 

Null  and  Alternate  Hypotheses,  FO  Number  1.  The  parameter 
of  interest  is  pi,  defined  as  the  population’s  mean  ECP  processing  time. 
The  null  hypothesis  is 

Ho:  px  <  p0  (2) 

and  the  alternate  hypothesis  is 

Ha:  Mj  >  p0  (3) 

where  p0  is  the  null  value  (the  standard)  of  147  days  based  on  the  CMP 
stated  time  for  no-cost  proposals  to  be  processed  from  TCP  receipt  to 
supplemental  agreement  (9:3-2).  This  alternate  hypothesis  calls  for  an 
upper-tailed  test  because  a  large  test  statistic  value,  which  represents 
the  value  above  which  the  population  mean  exists,  will  tend  to  refute 
the  null  hypothesis  (11:281).  Indeed,  if  the  LBV  is  higher  than  p0 , 
then  the  data  indicates  with  90%  certainty  that  the  population's 
processing  time  is  greater  than  the  standard  time  presented  in  the  CMP. 
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Thus  the  value  of  LBV  will  be  used  as  the  test  statistic  value  to  test 


the  null  hypothesis,  and  it  will  be  computed  using  Equation  1  above. 
Thus,  we  accept  the  null  hypothesis  when  the  lower  bound  value  of  the 
90%  one-sided  confidence  interval  is  less  than  or  equal  to  the  CMP 
standard  time,  and  we  reject  the  null  hypothesis  if  the  LBV  is  higher 
than  the  CMP  standard  time.  If  the  null  hypothesis  is  rejected,  then 
the  data  indicates  that  the  ECP  process  is  taking  more  time  than  the  CMP 
allows,  possibly  because  the  process  is  untimely. 

Answer  FO  Number  1.  The  calculated  LBV  will  be  used  to 
answer  the  focus  question.  Namely,  if  the  LBV  is  less  than  the  time 
stated  in  the  CMP,  then  the  answer  to  the  FQ  is  "No,  the  ACM  SPO  ECP 
process  is  not  taking  too  much  time."  However,  if  the  time  stated  in 
the  CMP  is  less  than  the  LBV,  then  the  answer  to  the  FQ  is  "yes."  A  bar 
chart  will  be  constructed  which  will  pictorially  show  the  answer  to  the 
focus  question.  As  an  example,  Figure  9  includes  two  vertical  bars;  the 
left  bar’s  height  represents  the  number  of  days  the  CMP  allows  for 
processing  no-cost  ECPs ,  and  the  right  bar's  height  represents  an 
example  of  the  mean  number  of  days  the  sample  ECPs  could  have  taken  to 
be  completed.  The  lower  bound  value  is  represented  by  the  lower  edge  of 
the  shaded  region.  Also,  the  relationship  between  the  LBV  and  the  top 
of  the  left  bar  provides  a  visual  answer  to  the  focus  question.  For 
example,  if  the  lower  edge  of  the  shaded  region  is  above  the  top  of  the 
left  bar,  then,  with  90%  confidence,  the  population  average  is  above  the 
standard  time  and  the  answer  to  FQ  Number  1  is  yes,  the  process  is 
indicated  as  untimely.  However,  remember  the  caveat  presented  above 
with  the  first  assumption:  the  answer  may  be  "yes,  the  process  is 
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EXAMPLE  OE  ANSWERING  THE  FQ 


untimely,"  but  the  answer  is  based  on  the  assumption  that  the  time 
requirements  stated  in  the  CMP  are  reasonable  and  achievable.  Now,  if 
the  null  hypothesis  is  rejected,  then  the  following  four  FQs  will  need 
to  be  answered. 

FQ  Number  2 .  Are  individual  process  steps  taking  too  much  time? 
A  positive  answer  to  this  question  will  tend  to  indicate  a  timeliness 
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problem  within  a  specific  processing  step.  This  FQ  will  be  asked  for 
each  ECP  process  step. 

Assumptions  for  FO  Number  2.  Seven  assumptions  will  be  made 
to  answer  this  focus  question. 

One .  The  first  assumption  is  that  the  CMP  is  an 
appropriate  measurement  tool  to  use  to  compare  the  timeliness  of  the  ECP 
process . 

Two .  The  second  assumption  states  that  the  collected 
data  is  representative  not  only  of  the  process  as  it  existed  in  the 
seven  month  time  window  from  which  the  data  was  collected  but  also  of 
the  process  from  the  end  of  that  seven  month  window  through  July  1990. 

Three  and  Four.  The  reports  from  which  the  data  was 
collected  are  both  accurate  and  current,  and  the  data  constitutes  a 
random  sample  from  the  population  of  all  ECPs  processed  by  the  ACM  SPO. 
This  sample  includes  all  ECPs  which  had  any  activity  within  the  last 
seven  months  cf  1989  regardless  of  cost,  disposition,  or  whether  or  not 
a  TCM  was  conducted. 

Five.  ftatistics,  including  the  sample  mean  a-  i 
sample  standard  deviation,  will  be  computed  using  the  set.  of  the  number 
of  days  each  process  step  took  for  each  ECP.  A  90%  one-sided  confidence 
interval  and  the  interval's  LBV  will  be  calculated  for  each  process 
step.  This  interval  will  he  calculated  differently  based  on  the  sample 
size.  If  the  sample  size  is  greater  than  30,  then  the  Central  Limit 
Theorem  will  be  invoked  and  the  sample  mean  will  therefore  have  an 
approximately  normal  distribution;  the  confidence  interval  then  will  be 
able  to  be  computed  using  the  z  distribution  (11:295). 
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Six . 


If  the  sample  size  is  less  than  30,  then  an 


assessment  will  be  made  to  determine  the  data's  distribution  shape.  A 
computer  program,  Statistix,  will  be  used  to  perform  a  Wilk-Shapiro  test 
of  normality,  producing  a  rankit  plot  of  the  variable  [process  time]  and 
computing  an  approximate  Wilk-Shapiro  normality  statistic  called  the 
Shapiro-Francia  statistic  (22:8.4).  "If  the  sample  conforms  to  a  normal 
distribution,  a  plot  of  the  rankits  against  the  order  statistics  should 
result  in  a  straight  line,  except  for  random  variation"  (22:8.5).  The 
sixth  assumption  is  that  if  the  data  produces  an  estimated  Wilk-Shapiro 
value  of  greater  than  0.90  and  if  the  rankits  plot  follows  a  linear  pat¬ 
tern,  then  the  data  will  be  assumed  to  follow  the  normal  distribution. 

Seven .  If,  however,  the  rankits  plot  is  not  close  to 

linear,  based  on  visual  inspection,  or  if  the  Shapiro- 'rancia  number  is 

below  0.90,  then  the  researcher  will  conduct  a  Kolmogorov- Smirnov  (K-S) 

test  to  determine  if  the  shape  of  the  distribution  is  exponential.  In 

regards  to  the  K-S  test's  applicability,  Law  and  Kelton  state 

. . .  the  original  form  of  the  K-S  test  is  valid  only  if  all  the 
parameters  of  the  hypothes-' zed  distribution  are  known;  i.e.,  the 
parameters  cannot  have  been  estimated  from  the  data.  In  recent 
years,  however,  the  K-S  test  has  been  extended  to  allow  for  es¬ 
timation  of  the  parameters  in  the  cases  of  normal,  exponential, 
and  Weibull  distributions.  (20:199) 

The  researcher  does  not  know  what  the  population  parameters  are,  so  the 
parameters  will  be  estimated.  These  estimates  will  be  used  to  determine 
the  adjusted  K-S  test  statistic  which  will  then  be  compared  to  critical 
values  (CVs) ,  given  by  Law  and  Kelton,  for  a  90%  confidence  factor.  For 
the  exponential  distribution,  the  CV  is  equal  to  0.990  (20:201).  If  the 
calculated  adjusted  test  statistic,  which  is  computed  according  to  the 
equations  presented  by  Law  and  Kelton,  is  less  than  the  CV,  then  the 
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test  will  have  indicated  a  good  fit  (20:201).  Thus  the  seventh 
assumption  is  that  if  the  adjusted  K-S  statistic  is  less  than  0.990, 
then  the  data  will  be  assumed  to  follow  the  exponential  distribution. 

Method  for  FQ  Number  2.  The  FC  will  be  answered  by 
comparing  the  actual  times  to  complete  each  individual  process  step  to 
the  amount  of  time  each  step  is  allowed  in  the  CMP.  The  comparison  will 
be  made  by  determining  the  lower  bound  value  of  a  90%  one-sided 
confidence  interval  about  the  means  of  the  data  and  seeing  whether  or 
not  the  LBV  is  higher  than  the  value  of  the  time  stated  in  the  CMP.  If 
the  LBV  is  higher  than  the  CMP  value,  then  the  data  will  tend  to 
indicate  an  untimely  process  step  (remember  the  first  assumption). 

Collect  the  Data.  The  data  that  will  be  collected  for 
each  individual  ECP  are  listed  in  Table  4.  Table  4  also  shows  what 
other  identifying  information  will  be  collected. 

Manipulate  the  Data.  For  each  ECP,  the  milestone 
dates  will  be  put  into  a  QUATTRO  PRO  spreadsheet  and  the 
"(3DATEVALUE ( string) "  function  will  be  used  to  convert  the  dates  into 
numerical  equivalents.  These  numerals  will  then  be  subtracted  to 
determine  the  number  of  days  that  were  required  to  complete  each  ECP 
process  step.  The  result  will  be  a  table  (in  the  spreadsheet) 
consisting  of  multiple  columns  of  numbers  that  represent  the  number  of 
days  the  ECFs  took  to  be  processed  at  each  process  step. 

Determine  the  Sample  Statistics.  Next,  QUATTRO  PRO's 
@AVG  and  @STDS  functions  will  be  used  to  determine  the  sample  mean  and 
sample  standard  deviation  of  the  data  for  each  process  step.  These  sta¬ 
tistics  will  be  used  to  determine  the  LBVs  of  a  90%  one-sided  confidence 
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TABLE  4 


DATA  THAT  WILL  BE  COLLECTED  TO  ANSWER  FQ  2 


Identifying  Information 

ECP  Number:  the  peculiar  number  for  each  ECP 

OPR's  Org  Symbol:  for  example,  "VCA"  for  Projects 

Cost  Information:  Proposed  cost  and  final  negotiated  cost 

Milestone  Events  for  Which  Dates  Will  Be  Collected 


TCP  Receipt: 
TCM : 

ECP  Receipt: 
CCB : 

GD/C  Response: 
OPR  Approval : 
Review  Comp: 
S/A: 


date  the  TCP  is  received  by  the  SPO 

date  the  TCM  is  conducted 

date  the  ECP  is  received  by  the  SPO 

date  the  proposal  is  briefed  to  the  CCB 

date  the  contractor  responds  to  the  CCB  comments 

date  the  OPR  approves  the  CCB- directed  updates 

contractual  review  process  completion  date 

date  the  change  is  formally  put  on  contract 


interval  to  the  numerically  lower  side  of  the  sample  mean.  However,  the 
value  of  the  LBV  depends  not  only  on  the  distribution  shape  of  the  data 
but  also  on  the  sample  size.  Three  combinations  of  distribution  shape 
and  sample  size  exist,  driving  a  requirement  to  determine  the  LBV  using 
three  different  equations.  These  three  combinations  are  discussed  now. 

Sample  Size  30  or  More.  When  the  sample  size  is 
30  or  more,  then  the  lower  bound  value  of  the  confidence  interval  is 
determined  using  Equation  1  above. 

Normal  Distribution.  Sample  Size  Less  Than  30 . 
When  the  sample  size  is  less  than  30  and  the  distribution  has  been 
determined  to  be  approximately  normal,  then  the  lower  bound  value  of  the 
confidence  interval  is  determined  using  Equation  4  below: 

LBV  -  MT  -  t(a , v)  *  (S//n)  (4) 
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where  t(a,v)  is  equal  to  the  t  critical  value  based  on  the  confidence 
factor  a  and  the  number  of  degrees  of  freedom  v,  where  v  =  n-1  (11:268). 
For  a  90%  one-sided  confidence  interval,  a  is  equal  to  0.10,  so  the  t 
critical  value  can  be  read  off  Table  A. 5  in  Devore's  text  (11:267). 

Exponential  Distribution.  When  the  shape  of  the 
distribution  cannot  be  declared  approximately  normal,  then  the  adjusted 
K-S  statistic  will  be  computed.  The  researcher  will  calculate  the  K-S 
statistic,  using  MathCAD  software,  in  accordance  with  the  method 
outlined  by  Law  and  Kelton  on  pages  199  through  203  (20:199-203).  If 
the  adjusted  K-S  statistic  confirms  that  the  data  is  approximately 
exponential,  then  the  lower  bound  value  of  the  90%  one-sided  confidence 
interval  will  be  calculated  using  Equation  5  below: 

LBV  -  (2  *  Sx4)  /  y2  (5) 

where  "Zx^"  is  the  sum  of  all  of  the  times  for  each  process  step  and  yz 
is  the  chi-squared  critical  value  based  on  the  number  of  elements  in  the 
data  set  and  the  desired  confidence  (11:257-258).  Table  A. 6  from 
Devore's  text  will  be  used  to  determine  the  chi-squared  number  for 
sample  sizes  less  then  20;  for  sample  sizes  greater  than  20  the  chi- 
squared  critical  value  will  be  estimated  (11:636). 

Null  and  Alternate  Hypotheses,  FQ  Number  2.  The  parameter 
of  interest  is  /u2,  defined  as  the  population's  mean  ECP  processing  time 
for  each  process  step.  The  null  hypothesis  for  each  step  is 

Ho:  n2  <  Mo  (6) 

and  the  alternate  hypothesis  is 

Ha:  M2  >  Mo  (7) 
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where  p0  is  the  null  value  (the  standard)  of  the  number  of  days  the  CMP 
states  the  process  step  should  take  to  be  completed.  The  null  values 
are  repeated  in  Table  5  below.  The  alternate  hypothesis  calls  for  an 


TABLE  5 

PROCESS  STEP  NULL  VALUES 


Number 

Number 

Number 

Number 

Process 

Step 

of  Days 

of  Days 

of  Days 

of  Days 

From 

_L 

To 

(No-Cost) 

( 0<0 . 5 ) 

( . 5<3 . 5) 

( 3 . 5<25 ) 

/ 

Submit  TCP 

Begin 

TCP 

/ 

Conduct  TCM 

14 

14 

14 

14 

TCM 

/ 

Submit  ECP 

38 

38 

38 

38 

ECP 

/ 

Conduct  CCB 

28 

28 

28 

28 

CCB 

/ 

Submit  CCB  Comments  7 

7 

7 

7 

Comments 

/ 

Submit  Response 

10 

10 

10 

10 

Response 

/  Complete  Review 

40 

85 

100 

115 

Review 

/ 

On  Contract 

10 

35 

40 

45 

Total  for  no-cost  proposal 

147 

217 

237 

257 

upper-tailed  test  because  a  large  test  statistic  value,  which  represents 
the  value  above  which  the  population  mean  exists,  will  tend  to  refute 
the  null  hypothesis  (11:281).  Indeed,  if  the  LBV  is  higher  than  /i0 , 
then  the  data  will  indicate  with  90%  certainty  that  the  population's 
processing  time  is  greater  than  the  standard  time  presented  in  the  CMP. 
Thus  the  test  statistic  value  that  will  be  used  to  test  the  null 
hypothesis  is  the  value  of  LBV,  computed  from  Equation  1,  Equation  4,  or 
Equation  5  above.  And,  we  accept  the  null  hypothesis  when  the  lower 
bound  value  of  the  90%  one-sided  confidence  interval  is  less  than  or 
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equal  to  the  CMP  standard  time,  and  we  reject  the  null  hypothesis  if  the 
LBV  is  higher  than  the  CMP  standard  time.  If  the  null  hypothesis  is 
rejected,  then  the  data  indicates  that  the  ECP  process  is  taking  more 
time  than  the  CMP  allows,  possibly  because  the  process  step  is  untimely. 

Answer  FQ  Number  2 .  Using  the  calculated  LBV,  the  answer  to 
the  FQ  for  each  process  step  can  be  determined.  So,  if  the  LBV  is  lower 
than  the  stated  CMP  time  for  the  specific  process  step,  then  the  answer 
to  the  FQ  is  "No,  this  specific  ECP  process  step  is  not  taking  too  much 
time."  However,  if  the  time  stated  in  the  CMP  is  less  than  the  LBV, 
then  the  answer  to  the  focus  question  is  "yes." 

FQ  Number  3 .  Are  ACM  SPO  ECPs  being  routed  through  the  same 
process  steps  more  times  than  once?  A  positive  answer  to  this  question 
could  be  indicative  of  inefficiencies  in  either  the  SPO's  TCP  and  TCM 
review  process  or  in  the  contractor's  ability  to  update  the  ECP 
according  to  the  requirements  established  by  the  personnel  at  the  TCM  or 
in  the  CCBD  Of  course,  a  positive  answer  to  the  focus  question  also 
could  mean  that  the  environment  is  causing  instability  in  process  such 
that  the  needs  change  between  the  time  the  ECP  is  submitted  to  the  SPO 
and  boarded  at  the  CCB. 

Assumptions  for  FQ  Number  3.  Three  assumptions  will  be  made 
to  answer  this  focus  question. 

One .  Although  the  CMP's  Change  Proposal  Flow  indi¬ 
cates  that  a  possibility  exists  of  having  to  change  the  ECP  based  on 
comments  from  the  CCB,  it  does  not  provide  any  time  requirements  within 
which  the  proposal  steps  should  be  re-accomplished  (cmp:3-2a).  The  CMP 
ideal  flow  is  shown  in  Figure  10;  it  is  identical  to  the  ACM  SPO  ECP 
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flow  presented  in  Figure  7  in  Chapter  II  except  the  OPR  Approval  step 
loop  in  Figure  7  is  omitted  from  Figure  10;  this  additional  loop  is  not 
included  because  those  steps  are  not  a  part  of  the  ideal  flow.  Thus, 
the  first  assumption  is  that  the  ideal  flow  is  as  shown  in  Figure  10; 
where  no  process  steps  need  to  be  performed  more  than  once.  Further¬ 
more,  this  ideal  flow  will  be  the  baseline  requirement  against  which  the 
actual  process  flow  frequencies  will  be  compared. 

Two.  Three,  and  Four.  The  second  assumption  is  that 
the  collected  data  is  representative  not  only  of  the  last  seven  month 
period  of  1989  but  also  of  the  period  up  to  and  including  July  31,  1990. 
The  third  assumption  is  that  the  data  will  represent  a  random  sample, 
and  the  fourth  assumption  is  that  the  reports  from  which  the  process 
flow  data  will  be  collected  are  both  accurate  and  current. 

Method  for  FQ  Number  3.  The  FQ  will  be  answered  by  deter¬ 
mining  the  frequency  at  which  ECPs  must  re-accomplish  certain  steps  in 
the  ECP  process.  The  frequencies  will  be  determined  by  dividing  the 
number  of  times  ECPs  must  re-accomplish  certain  process  steps  by  the 
total  number  of  ECPs  that  "arrive"  at  a  decision  point.  The  researcher 
will  study  two  decision  points  including  the  CCB,  which  determines  the 
disposition  of  the  ECP,  and  the  OPR  Approval  decision  point,  which  is 
where  the  OPR  determines  whether  or  not  the  ECP  was  updated  according  to 
the  CCBD.  If  the  frequencies  are  relatively  high,  then  the  data  will  be 
indicative  of  a  process  that  possibly  includes  a  lack  of  effort  on  the 
part  of  the  SPO  during  the  TCP  review  process  or  at  the  TCM,  or  a  lack 
of  ability  of  the  contractor  to  follow  through  with  the  requirements 
stated  in  the  CCBD  (perhaps  because  it  is  not  written  clearly) . 
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Figure  10.  Ideal  ECP  Flow  as  Presented  in  the  CMP  (9:3-2) 


Collect  the  Data.  The  data  that  will  be  collected 
are,  for  each  individual  ECP,  the  sequence  of  milestone  events  that  the 
ECP  went  through  before  it  completed  its  processing.  For  example,  the 


data  will  take  on  the  form  of  a  list  of  milestone  events  that  could 
include  the  following:  TCP  Receipt  -  TCM  -  ECP  -  CCB  -  ECP  -  OPR 
Approval  -  Review  -  S/A.  For  this  example,  the  ECP  process  flow 
included  the  receipt  in  the  SPO  of  a  TCP;  the  SPO  conducted  a  TCM;  the 
TCM  comments  and  recommended  changes,  if  any,  were  incorporated  into  the 
TCP  and  the  TCP  was  then  submitted  as  the  technical  portion  of  the  ECP 
(the  cost  volume,  for  a  cost  proposal,  also  will  be  prepared  ar.d  submit¬ 
ted  at  this  time);  the  SPO  and  the  other  organizations,  including  SAC, 
OCALC ,  and  MDMSC,  reviewed  the  ECP  and  provided  comments  to  the  OPR  who 
then  briefed  the  ECP  to  the  CCB;  comments  were  generated  in  the  CCB, 
included  in  the  CCBD,  and  sent  to  the  contractor  who  updated  the  ECP 
with  the  changes  and  resubmitted  the  ECP  to  the  SPO;  the  OPR  approved 
the  changed  ECP  as  submitted;  the  contractual  review  process  was  com¬ 
pleted;  and  the  ECP  was  made  an  official  contractual  change.  If  the 
process  flow  includes  an  OPR  Approval  step,  then  the  contractor's 
submission  of  the  ECP  required  changes  that  possibly  should  have  been 
found  and  corrected  during  the  TCP  Review  and  TCM  process  steps. 

Manipulate  the  Data.  For  each  ECP,  the  flow  path  will 
be  determined.  The  individual  flow  paths  will  then  be  compiled  onto 
flow  charts  which  will  indicate  all  possible  routes  the  chosen  ECPs  went 
through  to  complete  processing  and  how  often  those  routes  were  accessed. 

Determine  the  Sample  Statistics.  Next,  two  frequen¬ 
cies  will  be  computed.  The  first  frequency  is  computed  by  dividing  the 
number  of  times  ECPs  are  Approved  with  Comments  by  the  number  of  ECPs 
that  met  the  CCB.  This  frequency  will  tend  to  indicate  how  well  the  TCP 
Review  and  TCM  process  works,  since  it  is  assumed  that  if  the  TCP  Review 
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process  is  functioning  correctly  then  most  problems,  concerns,  and  typo¬ 
graphical  errors  will  be  addressed  and  corrected  at  the  TCM.  The  second 
frequency  that  will  be  computed  is  determined  by  dividing  the  number  of 
times  the  OPR  does  not  approve  the  ECP  submittal  at  the  OPR  Approval 
decision  point,  but  instead  requires  the  contractor  to  make  additional 
changes  to  the  ECP  before  it  is  approved,  by  the  number  of  ECPs  that 
reach  the  first  OPR  Approval  decision  point.  A  relatively  high  fre¬ 
quency  here  will  tend  to  indicate  the  possibility  of  inadequate  communi¬ 
cation  between  the  ACM  SPO  and  the  contractor  or  the  inability  of  the 
contractor  to  update  the  ECP  as  requested.  Based  on  the  researcher's 
subjective  evaluation  of  the  ECP  process  in  the  ACM  SPO,  plus  through 
the  researcher's  personal  experiences  and  informal  discussions  with  ACM 
SPO  personnel,  the  researcher  believes  that  the  state  of  the  process  in 
the  SPO  is  such  that  70%  or  more  of  the  ECPs  should  be  Approved  at  the 
CCB  the  first  time  the  ECP  is  presented  to  the  CCB.  To  expect  a  much 
higher  percentage  of  Approved  ECPs  would  be  an  unreasonable  expectation. 
However,  the  researcher  believes  that  at  least  80%  of  the  ECPs  that  go 
to  the  OPR  for  approval  should  be  approved,  without  having  to  be  sent 
back  to  the  contractor  for  additional  editing. 

Null  and  Alternate  Hypotheses,  FQ  Number  3.  The  parameters 
of  interest  are  pa  and  pb,  where  pa  is  defined  as  the  proportion  of  ECPs 
that  are  Approved  with  Comments  at  the  CCB  and  pb  is  defined  as  the  pro¬ 
portion  of  ECPs  that  the  OPR  does  not  approve  at  the  OPR  Approval  step. 
The  "test"  that  will  be  employed  here  is  to  compare  values  of  pa  and  pb 
to  the  researcher's  subjectively  determ\ned  percentages  of  0.30  and 
0.20,  respectively. 
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Answer  the  FQ.  Using  the  frequency  data,  the  answer  to  the 


FQ  will  be  determined.  Namely,  if  the  frequency  with  which  ECPs  get 
Approved  with  Comments  is  greater  than  0.30,  then  there  answer  to  the 
focus  question  will  be  "yes,  a  significant  percentage  of  ECPs  are  not 
following  the  ideal  process  flow."  Furthermore,  if  the  frequency  with 
which  ECPs  must  be  processed  through  the  OPR  Approval  step  is  greater 
than  0.20,  then  a  problem  is  indicated.  Thus,  if  the  values  of  pa  and 
pb  are  higher  than  the  subjectively  determined  values,  then  the  data 
will  tend  to  indicate  possible  problems  with  the  TCP  Review  and  TCM 
steps,  the  contractor's  ability  to  update  the  documents  as  requested,  or 
some  other  problems. 

FQ  Number  4.  Are  there  any  specific  ECP  process  flow  paths  that 
are  more  timely  than  the  CMP  requirements?  This  question  is  different 
from  the  first  FQ  because  the  answer  to  this  question  requires  the 
comparison  between  the  actual  times  for  groups  of  ECPs  that  share  in 
common  the  specific  flow  path  taken  and  the  amount  of  time  provided  in 
the  CMP  for  an  ECP  following  th.  path.  For  example,  ECPs  that  are 
disapproved  at  the  ECP  should  take  less  time  to  process  than  ECPs  that 
are  approved  because  disapproved  ECPs  typically  do  not  have  to  go 
through  the  Contractual  Review  process  steps. 

Assumptions  for  FQ  Number  4.  Five  assumptions  will  be  made 
to  answer  this  focus  question. 

One .  The  first  assumption  is  that  the  CMP  is  an 
appropriate  measurement  tool  to  use  to  compare  the  timeliness  of  the 
specific  ECP  process  flows.  Thus,  if  the  actual  process  times,  as 
determined  using  the  methods  presented  below,  are  less  than  those 
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spelled  out  in  the  CMP,  then  the  process  flow  will  not  be  considered 
untimely.  However,  if  the  actual  process  flow  times  are  longer  than  the 
CMP  times,  then  the  answer  to  the  FQ  will  be  "yes." 

Two,  Three,  and  Four.  The  second  assumption  is  that 
the  collected  data  will  be  representative  of  the  process  as  it  has 
existed  through  July  1990,  the  third  assumption  is  that  the  collected 
data  constitutes  a  random  sample,  and  the  fourth  assumption  is  that  the 
reports  from  which  the  process  flow  data  will  be  collected  are  accurate 
and  current. 

Five.  Statistics,  including  the  sample  mean  and 
sample  standard  deviation,  will  be  computed  using  the  set  of  the  number 
of  days  each  group  of  ECPs  took  to  be  processed.  A  90%  one-sided 
confidence  interval  will  be  calculated.  The  fifth  assumption  is  that 
the  collected  data  follows  the  normal  distribution.  An  assessment  of 
this  assumption  will  be  made  using  Statistix's  Wilk-Shapiro  test  of 
normality.  If  the  Shap: ro-Francia  statistic  is  higher  than  0.90  and  if 
the  rankits  plot  is  approximately  linear,  then  the  data  will  be 
determined  to  follow  the  normal  distribution  (22:8.5). 

Six .  If,  however,  the  distribution  cannot  be  assumed 
to  be  normal,  the  researcher  will  conduct  a  K-S  test  to  determine  if  the 
shape  of  the  distribution  is  exponential.  For  the  exponential  distribu¬ 
tion,  the  critical  value  is  equal  to  0.990  (20:201).  If  the  calculated 
adjusted  K-S  test  statistic  is  less  than  the  CV,  then  the  test  will 
indicate  a  good  fit  (20:201).  Thus  the  seventh  assumption  is  that  if 
the  adjusted  K-S  statistic  is  less  than  0.990,  then  the  data  will  be 
assumed  to  follow  the  exponential  distribution. 
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Method  for  FO  Number  4.  The  FQ  will  be  answered  by 
comparing  the  actual  times  to  complete  process  flows  to  the  amount  of 
time  allowed  in  the  CMP  for  ECPs  following  that  flow  path. 

Collect  the  Data.  The  data  that  needs  to  be  collected 
includes,  for  each  ECP,  the  ECP  number  and  the  dates  when  the  first  TCP 
or  the  first  ECP  (for  ECPs  that  do  not  have  a  TCP)  is  submitted  to  the 
ACM  SPO  and  when  the  ECP  completed  its  processing  by  either  being  put  on 
contract  or  by  notifying  the  contractor  that  the  ECP  has  been 
disapproved  at  the  CCB . 

Manipulate  the  Fata.  The  ECPs  will  be  grouped 
according  to  the  specific  path  that  the  ECPs  followed.  Nine  separate 
paths  will  be  considered.  Table  6  below  indicates  the  paths  that  will 
be  studied. 


TABLE  6 

PROCESS  FLOW  PATHS  AND  CMP  TIMES  TO  COMPLETE  (9:3-2) 


Path  Number  and  Process  Flow 

Number  of  Days 

Allowed.  CMP 

1. 

TCP -TCM- ECP- CCB -ECP -OPR- REV- S/A 

147 

2  . 

TCP -TCM- ECP -CCB -ECP -OPR -OPR -REV -S/A 

147 

3. 

TCP -ECP -CCB -ECP -OPR -REV -S/A 

147 

4. 

TCP -TCM- ECP -CCB -REV -S/A 

147 

5. 

TCP -TCM -ECP -CCB -COMM  OUT 

97 

6  . 

TCP - TCM - TCP - TCM - ECP - CCB -  -  -  COMPLETION 

147 

7. 

ECP-CCB- - -S/A 

95 

8  . 

MISCELLANEOUS 

95-147 

9. 

ENTRY - COMPLETION 

95-147 

Note : 

time  allowances  assume  the  ECP  is  a 

no-cost  change 
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1.  TCP -TCM -ECP -CCB -ECP -OPR -REV- S/A,  where  a  TCP  was 
received,  a  TCM  was  held,  the  ECP  was  submitted  and  then  boarded  at  the 
CCB,  the  proposal  was  Approved  with  Comments,  the  ECP  was  updated  and 
submitted  to  the  SPO,  the  OPR  approved  the  updates,  the  Review  process 
was  completed,  and  the  change  was  put  on  contract 

2.  TCP-TCM-ECP-CCB-ECP-OPR-OPR-REV-S/A ,  where  an  ECP 
followed  Path  1  but  had  to  have  a  second  OPR  approval  step.  In  other 
words,  after  the  CCB,  the  contractor  updated  the  ECP,  submitted  it  to 
the  SPO,  but  the  OPR  did  not  approve  the  contractor's  updates.  The  lack 
of  approval  forced  the  contractor  to  edit  the  ECP  and  submit  it  to  the 
OPR  for  the  second  time  after  the  CCB  before  the  ECP  could  progress  to 
the  Review  process  steps. 

3.  TCP-ECP-CCB-ECP- OPR - REV -  S/A ,  where  the  only 
difference  between  this  path  and  Path  1  is  that  the  TCM  was  waived. 

4.  TCP-TCM-ECP-CCB-REV-S/A;  this  path  represents  the 
ideal  flow,  where  the  ECP  is  approved  as  written  at  the  CCB. 

5.  TCP -TCM -ECP -CCB -COMM  OUT,  where  these  proposals 
were  disapproved  at  the  CCB.  The  COMM  OUT  date  is  recorded  after  the 
CCBD  is  approved  for  release  to  the  contractor. 

6.  TCP-TCM-TCP-TCM-ECP-CCB-- -COMPLETION,  where  these 
ECPs  cycled  through  two  TCPs  and  two  TCMs  and,  after  the  CCB,  followed 
various  routes  to  completion  (for  example,  multiple  additional 
presentations  to  the  CCB  or  multiple  OPR  Approval  steps  were  required) . 

7.  ECP-CCB - S/A,  where  these  ECPs  arrived  for  the 

first  time  in  the  SPO  in  the  form  of  an  ECP,  not  a  TCP.  After  the  CCB, 
these  ECPs  followed  different  routes  to  the  supplemental  agreement  step. 
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ECPs  that  are  classified  as  "Urgent"  or  "Emergency"  type  ECPs  would  be 
expected  to  follow  this  path  because  of  the  time  requirements  to  process 
these  types  of  ECPs  (6:16).  In  fact  DOD-STD-480A  stipulates  a  target 
of  IS  calendar  day  to  put  an  "Urgent"  proposal  on  contract  and  a  target 
of  only  24  hours  to  put  an  "Emergency"  proposal  on  contract  after  the 
proposal  is  received;  however, 

The  criticality  of  the  need  for  decision  will  dictate  the  actual 
processing  time  for  Emergency  and  Urgent  ECPs.  Emergency  and 
Urgent  ECPs  should  be  proposed  based  upon  the  above  targets  unless 
otherwise  agreed  to  between  the  contractor  and  the  procuring 
activity.  (6:16) 

8.  MISCELLANEOUS ,  where  the  remaining  ECPs  are  lumped 
together  to  produce  a  conglomerate  of  differing  flow  paths  and  times. 

9.  ENTRY - COMPLETION,  where  the  total  average  time 

for  all  ECPs,  regardless  of  the  path  taken,  are  lumped  together. 

Determine  the  Sample  Statistics.  Next,  QUATTRO  PRO'S 
"@AVG"  and  "(aSTDS"  functions  will  be  used  to  determine  the  sample  mean 
and  sample  standard  deviation  for  each  of  the  groups  of  data.  These 
statistics  will  be  used  to  determine  the  LBVs  of  90%  one-sided 
confidence  intervals  to  the  numerically  lower  side  of  the  sample  mean. 
However,  the  values  of  the  LBVs  depend  on  the  distribution  shape  of  the 
data.  Two  possible  distribution  shapes  exist  for  this  data,  normal  and 
exponential.  These  two  possible  shapes  drive  the  need  for  two  different 
methods  to  determine  the  LBV. 

Normal  Distribution.  When  the  distribution  of 
the  sample  has  been  determined  to  be  approximately  normal,  using 
Statistix's  normality  test  described  above,  then  the  lower  bound  value 
of  the  confidence  interval  is  determined  using  Equation  8  below: 
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LBV  =  MT  -  t(a,v)  *  (S/Jn)  (8) 

where  t(a,v)  is  equal  to  the  t  critical  value  based  on  the  confidence 
factor  a  and  the  number  of  degrees  of  freedom  v,  where  v  —  n  -  1 
(11:268).  For  a  90%  one-sided  confidence  interval,  a  is  equal  to  0.10, 
so  the  t  critical  value  can  be  read  off  Table  A. 5  in  Devore's  text 
(11:267)  . 

Exponential  Distribution.  When  the  shape  of  the 
distribution  cannot  be  declared  approximately  normal,  then  the  adjusted 
K-S  statistic  will  be  computed.  The  researcher  will  calculate  the  K-S 
statistic,  using  MathCAD  software,  in  accordance  with  the  method 
outlined  by  Law  and  Kelton  on  pages  199  through  203  (20:199-203).  If 
the  adjusted  K-S  statistic  confirms  that  the  data  is  approximately 
exponential,  then  the  lower  bound  value  of  the  90%  one-sided  confidence 
interval  will  be  calculated  using  Equation  9  below: 

LBV  -  (2  *  Ex^  /  yz  (9) 

where  "Ex1"  is  the  sum  of  all  of  the  times  for  each  process  step  and  yz 
is  the  chi-squared  critical  value  based  on  the  number  of  elements  in  the 
data  set  and  the  desired  confidence  (11:257-258).  Table  A. 6  from 
Devore's  text  will  be  used  to  determine  the  chi-squared  number  for 
sample  sizes  less  then  20;  for  sample  sizes  greater  than  20  the  chi- 
squared  critical  value  will  be  estimated  (11:636). 

Null  and  Alternate  Hypotheses,  FO  Number  h.  The  parameter 
of  interest  is  ,  defined  as  the  population's  mean  ECP  processing  time 
for  each  individual  process  flow.  The  null  hypothesis  for  each  flow  is 

Ho:  5  a*o  (10) 

and  the  alternate  hypothesis  is 
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Ha:  >  Mo  (11) 

where  n0  is  the  null  value  (the  standard)  of  the  number  of  days  the  CMP 


states  the  process  should  take  based  on  the  specific  process  steps 
included  in  the  flow  in  question.  The  null  values  are  repeated  in 
Table  6  above.  The  alternate  hypothesis  calls  for  an  upper- tailed  test 
because  a  large  test  statistic  value,  which  represents  the  value  above 
which  the  population  mean  exists,  will  tend  to  refute  the  null 
hypothesis  (11:281).  Indeed,  if  the  LBV  is  higher  than  /i0,  then  the 
data  will  indicate  with  90%  certainty  that  the  population's  processing 
time  is  greater  than  the  standard  time  presented  in  the  CMP.  Thus  the 
test  statistic  value  that  will  be  used  to  test  the  null  hypothesis  is 
the  value  of  LBV,  computed  from  Equation  8  or  Equation  9  above.  And,  we 
accept  the  null  hypothesis  when  the  lower  bound  value  of  the  90%  one¬ 
sided  confidence  interval  is  less  than  or  equal  to  the  CMP  standard 
time,  and  we  reject  the  null  hypothesis  if  the  LBV  is  higher  than  the 
CMP  standard  time.  If  the  null  hypothesis  is  rejected,  then  the  data 
indicates  that  the  ECP  process  along  that  flow  path  is  taking  more  time 
than  the  CMP  allows,  possibly  because  the  process  steps  along  that  path 
are  untimely  or  because  the  ECPs  in  that  group  had  to  have  specific 
process  steps  accomplished  more  than  one  time. 

Answer  FQ  Number  4.  Using  the  calculated  LBV,  which 
represents  the  lower  bound  above  which  the  population  means  are,  the 
answer  to  the  FQ  for  each  process  path  can  be  determined.  So,  if  the 
LBV  is  lower  than  the  stated  CMP  time  for  the  specific  process  flow 
path,  then  the  answer  to  the  FQ  is  "No,  this  specific  ECP  process  flow 
path  is  not  necessarily  taking  too  much  time."  However,  if  the  time 
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stated  in  the  CMP  is  less  than  the  LBV,  then  the  answer  to  the  focus 
question  is  "yes,  the  data  indicates  that  the  ECPs  that  follow  this 
process  flow  path  on  average  take  too  much  time." 

FO  Number  5 .  Is  the  number  of  open  ECPs  awaiting  processing  in 
any  particular  process  step  disproportionate  to  the  number  that  should 
be  waiting  in  that  step?  This  focus  question  attempts  to  determine 
whether  or  not  one  or  more  processes  are  untimely  to  the  point  that  a 
disproportionate  number  is  being  processed  in  that  step.  If,  for 
example,  95%  of  the  open  ECPs  are  waiting  to  be  boarded  at  the  CCB,  but 
only  20%  of  the  ECPs  should  be  waiting  processing  there,  then  a  possible 
problem  with  the  SPO  ECP  review  process  will  be  indicated. 

Assumptions  for  FO  Number  5.  Six  assumptions  will  be  made 
to  answer  this  focus  question. 

One .  If  the  amount  of  time  that  each  process  step 
takes  is  divided  by  the  total  processing  time,  and  if  the  CMP’s  times 
are  used,  then  the  expected  proportionate  time  each  step  should  take 
will  be  computed.  Furthermore,  if  ECPs  follow  the  ideal  flow,  and  if 
each  ECP  takes  the  amount  of  time  stated  in  the  CMP  to  complete  each 
process  step,  tren  the  number  of  open  ECPs  should  be  distributed  through 
the  process  steps  according  to  the  proportionate  time  each  step  should 
take  to  be  completed.  The  major  assumption  here  is  that  the  open  ECPs 
will  be  waiting  for  processing  in  the  process  steps  at  the  same 
proportion  as  the  expected  proportionate  amount  of  time  that  the  process 
step  should  take  to  be  completed.  These  expected  proportions  have  been 
computed  for  each  process  step,  and  they  are  shown  in  Table  7  along  with 
the  munber  of  days  the  process  steps  should  take  according  to  the  CMP. 
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TABLE  7 


EXPECTED  PROPORTIONATE  TIMES  FOR  PROCESS  STEPS  BASED  ON  CMP  (9:3-2) 


Step 

# 

Steo  From  /  To 

Number 
of  Davs 

Proportion 

Percentage 

1. 

TCP 

/  Submit  TCP 
/  Conduct  TCM 

Begin 

14 

9.5=  plo 

2. 

TCM 

/  Submit  ECP 

38 

25.9  =  p2o 

3. 

ECP 

/  Conduct  CCB 

28 

19.0  =  p3o 

4. 

CCB 

/  GD/C  Submit  Response 

17 

11.6  =  p4o 

5. 

Response 

/  Complete  Review 

40 

27.2  =  p5o 

6. 

Review 

/  On  Contract 

10 

6.8  =  p6o 

Total 

147 

100.0 

Furthermore,  the  process  steps  have  been  named  Step  1  through  Step  6, 
and  the  expected  proportions  have  been  defined  plo  through  p6o.  So  if 
this  first  assumption  is  valid,  then  27.2%  of  the  open  ECPs  should  be  in 
the  Contractual  Review  cycle.  If,  however,  60%  of  the  open  ECPs  are 
actually  in  the  contractual  review  cycle,  then  that  process  might  be 
indicated  as  a  potential  problem  area  which  may  need  further  study  to 
determine  why  the  high  percentage  of  open  ECPs  are  awaiting  completion 
within  that  process  step. 

Two.  Three,  and  Four.  The  second  assumption  is  that 
the  data  that  will  be  collected  are  representative  of  the  process  as  it 
has  existed  through  July  1990,  and  the  third  assumption  is  that  the 
collected  data  represents  a  random  sample.  Fourth,  the  reports  from 
which  the  data  will  be  collected  are  assumed  to  be  accurate  and  current. 

Five.  The  fifth  assumption  is  that  times  stated  in 
the  CMP  are  an  appropriate  measurement  tool  with  which  to  produce  the 


expected  proportionate  numbers  of  open  ECPs  in  each  of  the  process 
steps . 

Six .  Since  the  researcher  will  be  comparing 
proportions  where  the  category  probabilities  are  completely  specified, 
he  will  use  a  multinomial  experiment  and  conduct  a  chi -squared  goodness 
of  fit  test  (11:557).  The  experiment  will  consist  of  selecting  all  n 
open  GD/C  ECPs  from  the  population  of  all  open  ECPs  and  categorizing 
each  one  by  identifying  which  process  step  it  is  in,  Step  1  through  Step 
6.  This  experiment  will  be  approximately  multinomial  provided  that  n, 
the  number  of  open  ECPs  selected,  is  much  smaller  than  the  population 
size  (11:557).  The  researcher  assumes  that  the  number  of  open  ECPs 
selected  is  much  smaller  than  the  population  size. 

Method  for  FO  Number  5.  The  FQ  will  be  answered  by 
comparing  the  actual  number  of  ECPs  in  each  process  step  to  the  expected 
number  of  ECPs;  the  expected  number  is  equal  to  n  multiplied  by  the 
proportions  given  in  Table  6. 

Collect  the  Data.  The  researcher  will  collect,  on 
August  7,  1990  (a  date  that  new  ECP  reports  will  be  generated), 
information  on  all  open  GD/C  ECPs.  The  information  to  be  collected 
includes  a  statement  that  indicates  which  ECP  process  step  the  ECP  is 
in.  For  example,  if  the  ECP  is  being  reviewed  prior  to  its  first 
presentation  at  the  CCB,  then  the  researcher  will  record,  for  that  ECP, 
"Step  3,"  which  will  indicate  that  the  ECP  is  somewhere  in  the  ECP 
review  process . 

Manipulate  the  Data.  After  data  on  all  open  GD/C  ECPs 
is  collected,  the  researcher  will  sum  up  the  number  open  ECPs  that  are 
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in  each  of  the  process  steps.  For  example,  the  number  of  ECPs  in  the 
ECP  review  process  will  be  determined;  these  numbers  will  be  recorded  in 
a  table  under  the  heading  "Ni . " 

Determine  the  Sample  Statistics.  Next,  the  chi- 
squared  test  statistic  value  will  be  computed.  This  test  statistic 
value  will  be  computed  by  adding  up  each  step’s  contribution  to  the 
overall  test  statistic  value;  these  contributions  are  computed  using 
Equation  12 

y2i  “  (Ni-nPi)2/npi  (12) 

where  y2A  is  the  step's  contribution  to  the  chi-squared  test  statistic 
value,  Na  is  the  actual  number  of  open  ECPs  found  to  be  waiting  in  the 
ith  process  step,  n  is  the  total  sample  size,  and  npA  is  the  expected 
number  of  open  ECPs  in  each  ith  process  step  (11:560).  A  table  will  be 
provided  in  Chapter  IV  which  shows,  for  each  process  step,  the  actual 
number  (Nt)  ,  the  expected  number  (npA)  ,  the  step's  contribution  to  the 
chi-squared  test  statistic  (y2i)  ,  and  the  sum  of  the  individual  contri¬ 
butions  (labeled  "Total  y2  test  statistic  value").  The  value  of  y2  will 
then  be  compared  to  the  chi -squared  critical  value,  which  is  based  on 
the  significance  level,  a,  of  the  test  and  the  number  of  degrees  of 
freedom,  k  (11:559).  The  significance  level  will  be  set  to  0.10,  and 
k  is  equal  to  the  number  of  categories  minus  one  (11:559).  Thus,  for 
a  =  0.10  and  k  -  5 ,  the  chi-squared  critical  value  is  9.236  (11:636). 

Null  and  Alternate  Hypotheses,  F0  Number  5.  The  parameters 
of  interest  are  the  actual  proportions  of  ECPs  waiting  processing  in  the 
six  ECP  process  steps.  These  proportions  are  labeled  pi  through  p6 ,  and 
the  null  hypothesis  with  respect  to  these  proportions  is  expressed  below 
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Ho:  pi  -  plo,  p2  -  p2o,  .  .  .  ,  p6  -  p6o  (13) 

where  plo  through  p6o  represent  the  null  proportion  values,  as  shown  in 
Table  7  above  (11:559).  The  alternate  hypothesis  is  expressed  as 

Ha:  at  least  one  pi  does  not  equal  pio  (14) 

where  pi  represents  pi  through  p6  and  pio  represents  plo  through  p6o 
(11:559).  The  test  statistic  value  is  computed  by  using  Equation  12 
above  and  summing  the  individual  step  contributions  to  produce  the  chi- 
squared  test  statistic  value.  This  test  statistic  value  is  then 
compared  to  the  chi-squared  critical  value,  which  was  found  to  be  equal 
to  9.236  (11:636).  The  answer  to  FQ  5  will  be  based  on  whether  or  not 
the  null  hypothesis  is  rejected.  The  null  hypothesis  will  be  rejected 
if  the  computed  chi-squared  test  statistic  value  is  greater  than  or 
equal  to  the  critical  value  of  9.236,  which  will  produce  an  answer  with 
significance  level  0.10  (11:559,636).  Armed  with  this  information,  FQ 
Number  5  will  be  able  to  be  answered. 

Answer  the  FQ.  So,  the  focus  question  will  be  answered  by 
comparing  the  calculated  chi-squared  test  statistic  value  to  the  chi- 
squared  critical  value.  If  the  test  statistic  value  exceeds  9.236,  then 
the  null  hypothesis  will  be  rejected,  and  the  answer  to  FQ  5  will  be 
"yes,  the  number  of  open  ECPs  awaiting  processing  in  at  least  one 
process  step  is  disproportionate  to  the  number  that  should  be  waiting  in 
that  step."  However,  if  the  test  statistic  value  is  less  than  9.236, 
then  the  answer  will  be  "no,  the  data  does  not  indicate  a  dispropor¬ 
tionate  number  of  ECPs  awaiting  processing  in  any  particular  process 
step."  Furthermore,  regardless  of  the  answer,  this  technique  will 
indicate  which  process  steps  have  the  most  disproportionate  number  of 
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ECPs  awaiting  processing  by  virtue  of  the  magnitude  of  the  individual 
process  step's  contribution  to  the  chi-squared  test  statistic  value. 

Summary  of  Efforts  to  Determine  Timeliness.  This  section 
presented  five  focus  questions  which,  when  answered,  are  intended  to 
show  the  overall  state  of  the  ECP  process,  whether  good  or  bad, 
especially  with  regard  to  the  process'  timeliness.  The  five  focus 
questions  will  provide  increasingly  detailed  evidence  that  will,  taken 
as  a  whole,  convince  the  reader  that  the  ACM  SPO's  ECP  process  is  either 
timely  or  untimely  based  on  the  assumption  that  the  processing  times 
presented  in  the  CMP  are  reasonable.  If  the  result  of  the  above 
methodology  shows  that  the  process  is  untimely,  then  the  assumption 
about  the  goodness  of  the  times  presented  in  the  CMP  will  need  to  be 
validated.  The  goal  of  the  methodology  presented  below  is  to  validate 
that  assumption.  If  the  process  is  found  to  be  untimely,  and  if  the 
methodology  below  shows  that  the  problem  is  not  a  too-optimistic  CMP, 
then  the  conclusion  that  will  be  made  is  that  the  ACM  SPO  ECP  process  is 
untimely.  Furthermore,  the  results  of  both  methodologies  will  be  used 
to  prepare  recommended  courses  of  action  that  will  possibly  improve  the 
SPO's  ECP  processing  time. 

Analyze  the  Survey  Questionnaire  Results. 

Introduction .  The  survey  performed  by  the  CPIT  was  intended  to 
provide  the  team  with  some  data  about  their  current  ECP  process  (23) . 

The  survey  included  29  questions  which  the  respondent  was  to  answer  by 
circling  a  number  from  1  to  6 .  These  numbers  represented  responses  that 
the  respondent  could  use  to  either  agree  or  disagree  with  the  specific 
statement  at  hand.  The  possible  responses  were: 
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1  -  This  [statement]  is  almost  always  or  always  false 

2  -  This  is  false  most  of  the  time 

3  -  This  is  true  as  often  as  it  is  not  true 

4  -  This  is  true  most  of  the  time 

5  -  This  is  almost  always  or  always  true 

6  -  This  does  not  apply  to  me 

This  form  of  a  Likert  scale  provides  data  which  will  be  used  to  find  out 
whether  the  respondents  are  more  or  less  favorable  to  12  specific 
questions;  however,  it  cannot  be  used  to  indicate  how  strongly  the 
respondents  feel  about  the  answers  to  the  questions  (13:258).  The  12 
questions  will  be  selected  by  the  researcher  to  test  the  four  second- 
area  hypothesis  claims  made  in  Chapter  I,  and  as  indicated  in  Figure  11. 
These  four  hypothesis  claims  will  be  converted  into  four  focus 
questions,  and  the  12  survey  question  responses  will  tend  to  indicate  a 
positive  or  negative  response  to  each  focus  question.  The  hypotheses  to 
be  tested  are  restated  below,  along  with  the  specific  survey  questions 
that  will  be  used  to  answer  the  focus  questions.  Assumptions  will  then 
be  presented,  followed  by  the  methodology  that  will  be  used  to  analyze 
each  of  the  12  survey  questions.  A  technique  will  then  be  presented 
which  will  be  used  to  provide  an  objective  answer  to  the  focus 
questions.  Finally,  a  table  will  be  presented  that  shows  what 
information  will  be  obtained  by  completing  the  steps  outlined  in  the 
method  portion  of  this  section. 

Develop  Focus  Questions.  For  each  of  the  second-area  Chapter  I 
hypotheses,  the  focus  question  will  be  developed.  The  survey  questions 
that  will  be  analyzed  to  determine  the  answers  to  the  focus  questions 
will  then  be  presented. 

Communication .  The  first  hypothesis  stated  in  Chapter  I  is: 
ACM  SPO  personnel  do  not  sufficiently  communicate  with  other  ACM  program 
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personnel  within  or  outside  the  SPO.  The  focus  question  is:  Does  the 
data  indicate  that  people  in  the  SPO  adequately  communicate  with  one 
another  and  with  those  outside  the  SPO?  The  CPIT's  survey  included  five 
questions  regarding  communication.  These  five  survey  questions  will  be 
analyzed  to  determine  the  answer  to  the  focus  question. 
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One .  "I  [the  reviewer]  go  to  the  OPR  to  get  ir.y  ques¬ 
tions  answered  regarding  an  ECP/CCP  [Contract  Change  Proposal],  prior  to 
submitting  my  written  input."  If  the  survey  respondents  circle,  on 
average,  one  of  the  first  two  responses,  then  the  survey  would  tend  to 
indicate  that  reviewers  do  not  ask  the  OPR  questions  before  submitting 
their  written  responses  and  that  the  reviewers  do  not  orally  communicate 
with  the  OPRs . 

Two .  "I  am  informed  of  concessi  ns/changes  made  to  an 
ECP/CCP  during  negotiations  that  impact  me."  For  this  question,  an 
average  response  of  less  than  three  will  tend  to  support  the  assertion 
that  contracts  personnel  do  not  sufficiently  communicate  with  the  other 
personnel  involved  with  the  ECP,  especially  in  regards  to  changes  in  the 
scope  of  the  ECP  that  were  agreed  to  by  the  contracts  personnel  during 
the  negotiation  process. 

Three .  "I  [the  reviewer]  give  the  OPR  an  opportunity 
to  resolve  any  issues  I  have  prior  to  the  CCB."  Here,  an  average 
response  of  less  than  three  will  tend  to  support  the  assertion  that 
reviewers  do  not  sufficiently  communicate  with  the  OPRs. 

Four.  "I  [the  OPR]  am  aware  of  all  issues  prior  to 
them  being  raised  at  the  CCB  regarding  my  ECP/CCPs . "  For  this  question, 
an  average  response  of  less  than  three  will  tend  to  indicate  that  the 
reviewers  do  not  sufficiently  communicate  with  the  OPRs  before  the  CCB 
is  conducted. 

Five.  "The  CCB  Representatives  ask  questions  at  the 
CCB  that  have  already  been  resolved  with  their  working  people."  / 
average  response  of  more  than  i  ree  will  tend  to  indicate  that 
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functional  personnel  do  not  adequately  communicate  with  their  functional 


representatives  to  the  CCB. 

ECP  Submissions.  The  second  hypothesis  states  that  the 
contractor's  ECP  submissions  are  unclear,  lack  sufficient  detail,  and 
are  inaccurate.  The  focus  question  is:  Are  the  contractor’s  ECP 
submissions  clear,  provide  the  necessary  detail,  and  accurate?  The  CPIT 
survey  included  three  questions  which,  when  analyzed,  will  tend  to 
support  a  positive  or  negative  answer  to  the  focus  question. 

Six.  "The  Contractor  has  adequately  addressed  and 
documented  the  ECP/CCP  prior  to  submitting  it  to  the  Government."  A 
response  of  less  than  three  will  tend  to  indicate  the  contractor's  ECP 
submissions  lack  sufficient  detail. 

Seven.  "The  Contractor  complies  with  the  intent  of 
the  TCM  minutes."  Since  the  TCM  minutes  are  the  final  official  comments 
to  the  contractor  before  the  contractor  submits  the  ECP,  noncompliance 
with  those  minutes  indicates  an  inability  to  adequately  prepare  the  ECP. 
An  average  response  of  less  than  three  will  tend  to  indicate  that  the 
contractor's  ECP  submissions  are  inaccurate  or  lack  sufficient  detail. 

Eight .  "The  contractor  complies  with  the  intent  of 
the  CCB  Directive."  If  the  average  response  of  this  survey  question  is 
less  than  three,  then  the  indication  is  that  the  contractor's  ECP 
submissions  are  unclear,  lack  detail,  or  are  inaccurate. 

Personnel  Resources.  The  third  hypothesis  states  that  the 
ACM  SPO  lacks  adequate  personnel  resources  to  handle  the  number  of  ECPs 
in  the  ECP  process  at  any  one  time.  The  focus  question  is:  Does  the 
ACM  SPO  have  adequate  personnel  resources,  including  the  proper  numbers 
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wich  proper  training,  to  handle  the  number  of  ECPs?  The  survey  included 
two  questions  related  to  personnel  resources  or  the  time  the  individuals 
had  or  took  to  work  on  the  proposals. 

Nine.  "I  have  adequate  time  to  review  ECP/CCPs  and 
provide  comments  by  the  date  requested."  An  average  answer  less  than 
three  will  tend  to  indicate  that  personnel  feel  as  though  they  lack  the 
time  to  adequately  devote  to  ECPs,  perhaps  because  their  other  duties 
take  too  much  of  their  time. 

Ten.  "I  have  to  go  to  the  functionals  and  personally 
ask  for  their  comments  in  order  to  compile  them  for  CCB."  An  average 
answer  higher  than  three  would  tend  to  indicate  that  either  personnel 
simply  do  not  take  the  time  to  do  their  work,  the  ECPs  are  not  being 
handed  to  the  reviewers  by  their  organization  in  a  timely  manner,  or  the 
personnel  are  overloaded  with  other  work  so  that  they  do  not  have  the 
time  to  prepare  comments  in  a  timely  manner. 

Preparatory  Work  at  TCMs .  Lastly,  the  fourth  hypothesis 
stated  that  insufficient  up-front  effort  by  government  personnel  at 
Technical  Coordination  Meetings  (TCMs)  also  results  in  time  delays.  The 
focus  question  is:  Do  ACM  SPO  personnel  make  the  effort  to  review  the 
TCP  before  the  TCM  is  held?  The  survey  had  two  questions  related  to  the 
level  of  effort  individuals  applied  toward  preparing  for  TCMs. 

Eleven .  "I  attend  the  TCM  for  all  TCPs  that  I  would 
comment  on  as  ECPs/CCPs . "  Any  responses  below  three  would  tend  to 
indicate  that  either  time  or  schedule  constraints  prevent  the  person 
from  attending  the  TCM  or  that  the  person  just  does  not  believe  that  it 
is  teneficial  to  take  the  time  to  attend  the  TCM. 
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Twelve.  "I  receive  comments  from  functionals  on 


ECPs/CCPs  who  did  not  attend  the  TCM  where  they  [the  questions]  could 
have  been  resolved."  For  this  question,  an  average  answer  of  more  than 
three  would  indicate  that  the  OPRs  feel  that  personnel  are  not  adequate¬ 
ly  reviewirg  the  TCPs  prior  to  the  TCM. 

The  responses  to  these  12  survey  questions  will  be  analyzed  by 
determining  whether  the  total  average  responses  tend  to  indicate  either 
a  definitely  positive  or  a  definitely  negative  response  to  the  focus 
questions  and  determining  whether  or  not  general  agreement  exists  across 
the  SPO  regarding  the  specific  numerical  response  to  the  question. 

Then,  a  test  will  be  presented  which  will  provide  an  objective  determi¬ 
nation  of  whether  or  not  the  response  data  indicates  a  "yes"  or  a  "no" 
answer  for  each  of  the  four  focus  questions.  Now,  the  researcher's 
assumptions  will  be  stated. 

Assumptions .  Five  assumptions  will  be  stated  here. 

One.  The  first  assumption  is  that  the  survey  data  will  be 
able  to  provide  an  indication  of  ECP  process  problems.  The  survey 
questions  were  developed  by  the  CPIT  personnel  to  measure  the  percep¬ 
tions  that  SPO  personnel  had  about  the  ECP  process.  The  survey  was  not 
necessarily  prepared  to  minimize  problems  with  validity.  For  example, 
the  survey  does  not  necessarily  have  all  of  the  needed  control  questions 
to  make  sure  responses  are  not  driven  by  the  questions  (5).  However, 
the  assumption  employed  here  is  that  the  survey  will  be  useful  to 
provide  an  indication  of  ECP  process  problems. 

Two .  When  determining  the  overall  response  below,  one- 
tailed  confidence  intervals  will  be  constructed  about  the  mean  responses 


100 


of  Che  survey  questions.  The  second  assumption  is  that  more  than  30 
people  will  respond  to  each  question.  With  this  being  the  case,  the 
Central  Limit  Theorem  will  be  invoked  so  that  the  researcher  can  use  the 
z  distribution  to  calculate  the  upper  and  lower  bound  values  of  the 
intervals . 

Three .  Another  assumption  is  that  the  survey  respondents 
are,  on  average,  knowledgeable  enough  not  only  about  the  process  that 
exists  in  the  SPO  but  also  about  how  the  process  should  work. 

Four .  For  determining  the  consensus,  the  sample  sizes  will 
be  less  than  30.  However,  the  responses  will  be  assumed  to  follow  the 
normal  distribution.  This  assumption  will  be  verified  by  calculating 
Statistix's  Shapiro-Francia  statistic  and  by  observing  the  rankits  plot 
of  the  data.  The  assumption  will  be  assessed  by  comparing  this 
statistic  to  G.90  and  by  visually  observing  an  approximately  linear 
rankits  plot. 

Five.  The  final  assumption  is  that  the  answers  to  the 
survey  will  provide  interval  level  data  which  can  then  be  manipulated  to 
calculate  sample  means  and  standard  deviations. 

Method.  Eighty-six  people  provided  responses  to  the  survey 
questionnaire.  For  each  of  the  29  questions  in  the  survey,  the 
researcher  will  compile  the  responses  and  compute  statistics  needed  to 
measure  the  SPO's  perceptions  about  the  ECP  process. 

Collect  the  Data.  For  each  of  the  86  survey  responses,  the 
researcher  will  enter  the  answers  for  each  question  into  a  computer 
spreadsheet,  creating  a  table  of  data  with  86  rows  and  29  columns. 
Answers  of  "6"  will  not  be  included  in  the  table  because  these  responses 
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will  skew  the  averages  of  the  responses  which  are  needed  for  the  proce¬ 
dures  below. 

Manipulate  the  Data.  Two  steps  are  described  below: 
determining  the  overall  average  responses  to  the  12  selected  survey 
questions  and  determining  whether  a  general  consensus  exists  about  the 
specific  value  of  the  responses. 

Determine  the  Overall  Response.  To  provide  a  picture 
of  where  the  average  responses  fall  with  respect  to  the  value  of  three. 
90%  one-tailed  confidence  intervals  will  be  constructed  about  the  12 
question’s  mean  responses  so  that  an  assessment  can  be  made  regarding 
whether  or  not  the  population  mean  is  definitely  above  or  below  the 
value  of  three,  thereby  indicating  a  definitely  positive  or  negative 
response  to  the  survey  question.  These  intervals  will  have  either  a 
lower  bound  value  (LBV)  or  an  upper  bound  value  (UBV) ,  depending  on  the 
response  needed  to  support  a  "no"  answer  to  the  focus  question  (and 
thereby  supporting  the  hypothesis  claim).  For  example,  the  ninth  survey 
question,  "I  have  adequate  time  to  review  ECPs , "  needs  an  average 
response  of  less  than  three  (the  people  who  answered  the  question  need 
to  on  average  feel  that  the  answer  is  false)  to  support  a  no  answer  to 
the  third  focus  question  which  asks,  "Does  the  SPO  have  adequate 
personnel  resources?"  The  researcher  is  looking  for  an  average  response 
of  less  than  three,  so  the  researcher  will  calculate  an  UBV  below  which 
the  population  response  lies,  with  90%  confidence.  On  the  other  hand, 
the  tenth  survey  question,  "I  have  to  go  to  the  functionals  to  get 
responses,"  requires  an  average  response  of  greater  than  three  to 
support  a  no  answer  to  the  third  focus  question.  In  this  case,  a  LBV 
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will  be  calculated  above  which  the  population  response  lies,  with  90% 
assurance.  Since  the  sample  size  is  assumed  to  be  greater  than  30, 
Equations  15  and  16  below  use  the  z  distribution  to  calculate  the  LBV 
and  the  UBV 

LBV  -  MR  -  Za  *  (S/Jn)  (15) 

UBV  -  MR  +  Za  *  (S/Jn)  (16) 

where  MR  is  the  mean  response,  Zq  =  1.282,  and  n  is  the  sample  size 
(11:634).  Thus,  if  the  LBV  is  above  the  value  of  three,  or  if  the  UBV 
is  below  the  value  of  three,  then  support  for  the  hypothesis  claim,  from 
which  the  focus  question  was  derived,  will  be  indicated.  These  bound 
values  will  be  computed  and  bar  charts  will  be  constructed  to  show  where 
the  data  indicates  that  the  true  average  response  lies.  If  the  total 
confidence  interval  is  to  one  side  or  the  other  of  the  value  of  three, 
then  the  researcher  will  claim  that  the  data  tends  to  support  a 
definitely  positive  or  negative  response  to  the  focus  question.  A 
statement  will  be  made  regarding  whether  or  not  the  answer  to  the  focus 
question  tends  to  support  the  claim.  In  summary,  the  parameter  of 
interest  is  the  population  mean  response  to  each  of  the  survey  questions 
selected  by  the  researcher  to  test  the  major  hypothesis  claims.  The 
null  hypothesis  is  that  the  population  mean  is  equal  to  the  value  of 
three,  as  indicated  in  Equation  17  below.  The  alternate  hypothesis  will 
be  constructed  one  of  two  possible  ways,  as  Equation  18  and  Equation  19 
indicate . 


Ho 

:  Mr  -  3 

(17) 

Ha: 

Mr  > 

3,  need 

a  LBV 

(18) 

Ha: 

Mr  < 

3 ,  need 

a  UBV 

(19) 
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where  pR  is  the  parameter  of  interest  defined  as  the  population  mean 
response  to  the  survey  question,  "need  a  LBV"  implies  that  the 
population  response  needs  to  be  greater  than  three  to  indicate  a  "no" 
response  to  the  focus  question,  and  "need  a  UBV"  implies  that  the 
population  response  needs  to  be  less  than  three  to  indicate  a  "no" 
response  to  the  focus  question.  The  test  will  be  conducted  by 
calculating  the  appropriate  bound  value  and  seeing  whether  or  not  the 
LBV  is  greater  than  3  or  the  UBV  is  less  than  3.  If  yes,  then  the  null 
hypothesis  will  be  rejected  in  favor  of  the  alternate  hypothesis,  the 
focus  question  will  be  answered  "no,"  and  support  for  the  major 
hypothesis  claim  will  be  indicated.  The  next  step  is  to  see  if  positive 
consensus  exists  about  the  average  numerical  response. 

Determine  Consensus.  By  following  the  method  above, 
confidence  intervals  will  indicate  where  the  population  means  for  the 
responses  lie  with  respect  to  the  value  of  three.  These  confidence 
intervals  will  provide  one  piece  of  information  for  each  survey 
question.  The  sample  mean  will  provide  a  second  piece  of  information. 
However,  the  researcher  needs  more  information  so  that  a  point-measuring 
scheme  can  he  used  to  evaluate  the  net  effect  of  the  responses  to  the 
survey  questions  on  the  answers  to  the  four  focus  questions.  One 
additional  piece  of  information  that  would  be  useful  for  this  effort  is 
a  determination  of  whether  or  not  the  response  to  the  question  is 
consistent  regardless  of  the  general  attitudes  of  the  people  responding 
to  the  question.  In  other  words,  if  people  who  exhibit  an  overall 
favorable  attitude  tend  to  answer  a  question  negatively,  and  if  the 
people  who  exhibit  an  overall  unfavorable  attitude  also  tend  to  answer 


104 


that  question  negatively,  then  consensus  about  that  question  will  be 
assumed  to  exist  (13:256).  To  determine  whether  or  not  consensus  exists 
for  each  survey  question,  Each  respondent’s  total  average  score  will  be 
computed  using  QUATTRO  PRO'S  @AVG  function.  These  averages  will  be 
sorted  from  lowest  to  highest  using  QUATTRO  PRO'S  sorting  capability. 
Thus  the  respondents  having  the  lowest  average  scores  will  be  at  the  top 
of  the  list  and  the  respondents  with  the  highest  average  scores  will  be 
at  the  bottom  of  the  list.  Next,  the  top  28  sets  of  responses  on  the 
sorted  list  (the  "Low  Group,"  named  because  their  average  responses  were 
relatively  numerically  lower)  and  the  bottom  28  sets  of  responses  (the 
"High  Group")  of  the  list  will  be  selected.  The  two  group's  average 
responses  for  each  question  will  be  computed  so  that  the  two  extreme 
groups'  responses  can  be  compared  to  determine  whether  or  not  agreement 
exists,  among  those  who  responded,  about  the  responses  to  the  specific 
questions  (13:256).  Next,  for  each  of  the  12  individual  questions 
selected  by  the  researcher,  the  two  group's  means,  sample  standard 
deviations,  and  the  numbers  of  people  in  that  group  who  responded  to 
that  particular  question  will  be  computed  (13:257).  The  group  means  for 
these  12  questions  will  then  be  tested  with  a  pooled  two-sample  t  test 
to  determine  whether  or  not  the  groups'  mean  responses  are  the  same.  If 
the  data  indicates  that  the  mean  responses  are  the  same,  then  consensus 
will  be  said  to  exist.  This  technique  will  allow  the  researcher  to 
determine  whether  or  not  the  two  groups  have  the  same  attitude 
concerning  the  specific  question  (13:256).  The  pooled  two-sample  t  test 
will  be  conducted  by  calculating  the  test  statistic  value  using 
Equation  20  below  and  comparing  that  value  to  the  t  critical  value  which 
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is  based  on  the  significance  level  and  the  number  of  degrees  of  freedom 
(a  function  of  the  number  who  responded  to  the  question)  (11:336).  The 
significance  level  will  be  set  to  0.10,  and  the  t  critical  value  will  be 
read  off  of  Table  A. 5  in  Devore's  text  (11:635).  Equation  20,  obtained 
from  page  336  in  Devore's  text,  is: 

t  -  (MHG  -  MLG)/(Sp  *  J (1/m  +  1/n))  (20) 

where  t  is  the  test  statistic  value,  MHG  is  the  mean  response  from  the 
High  Group,  MLG  is  the  Low  Group's  mean  response,  Sp  is  the  pooled 
sample  standard  deviation,  m  is  the  High  Group's  sample  size,  and  n  is 
the  Low  Group's  sample  size  (11:336).  The  pooled  sample  standard 
deviation  is  determined  using  Equation  21  (11:335). 

Sp2  -  { (m- 1)*SHG2  +  (n- 1)*SLG2 )  /  (m+n-2)  (21) 

where  SHG  is  the  High  Group's  sample  standard  deviation  and  SLG  is  the 
Low  Group's  sample  standard  deviation.  For  each  question,  a  table  will 
be  provided  which  shows  whether  or  not  the  null  hypothesis  was  accepted 
or  rejected,  where  the  null  and  alternate  hypotheses  are 


Ho 

:  MHG  =  MLG 

(22) 

Ha: 

MHG  =/=  MLG. 

(23) 

So,  if  the  absolute  value  of  the  test  statistic  value  is  lower  than  the 
t  critical  value,  then  the  null  hypothesis  will  be  accepted,  supporting 
the  assertion  that  general  agreement  exists  about  the  numerical  value  of 
the  average  response  (11:336). 

Summary  Table  of  Collected  Information.  The  methodology 
described  above  will  produce  the  information  described  here.  For  each 
of  the  survey  questions  that  will  be  used  to  answer  the  four  focus  ques¬ 
tions,  the  needed  response  to  indicate  support  for  the  hypothesis  claim 
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TABLE  8 


SUMMARY  OF  INFORMATION  TO  BE  OBTAINED  FROM  METHODOLOGY 


MR  Needed 
to  Support 
Claim 


Focus  Question  I 

SQ  1  MR  <  3 

SQ  2  MR  <  3 

SQ  3  MR  <  3 

SQ  4  MR  <  3 

SQ  5  MR  >  3 

Focus  Question  II 

SQ  6  MR  <  3 

SQ  7  MR  <  3 

SQ  8  MR  <  3 

Focus  Question  III 

SQ  9  MR  <  3 

SQ  10  MR  >  3 

Focus  Question  IV 

SQ  11  MR  <  3 

SQ  12  MR  >  3 


Means 

LGM/HGM/Tot 


Response 

Consensus?  Definite? 


Values 


Yes 

Yes 

or 

or 

No 

No 

is  listed.  Other  data  to  be  collected  includes  the  value  of  the  average 
response,  an  indication  of  whether  or  not  the  population  mean  response 
is  to  the  "correct"  side  of  the  value  of  three  (thereby  supporting  a 
"no"  answer  to  the  focus  question),  the  values  of  the  High  Group  and  Low 
Group  means,  and  an  indication  of  whether  or  not  the  both  groups  felt 
similarly  about  the  answer  to  the  question.  This  information  will  be 
collected  and  provided  in  a  table  identical  to  the  one  shown  in  Table  8. 
Moreover,  a  bar  chart  will  be  prepared  to  show  not  only  the  two  groups' 
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(the  Low  Group  and  the  High  Group)  responses  but  also  the  mean  response 
of  the  total  data  set.  Another  bar  chart  will  be  prepared  that  shows 
the  results  of  computing  the  confidence  interval  about  the  total  data 
set  means  for  each  question.  This  chart  will  have  bars  with  shaded 
regions  which  represent  the  interval  within  which  the  true  mean  is 
located,  with  90%  confidence.  If  the  interval  does  not  cross  the  value 
of  three,  then  the  researcher  will  make  a  definite  statement  about  what 
the  data  indicates,  especially  in  light  of  the  other  reported 
information. 

Evaluating  the  Major  Hypotheses.  Unfortunately,  the  method 
presented  thus  far  does  not  lend  itself  to  objective  conclusions 
regarding  the  major  hypotheses  presented  in  Chapter  I.  For  example, 
recall  that  five  survey  questions  were  selected  to  test  the  hypothesis 
that  poor  communication  in  the  SPO  was  causing  untimely  ECP  processing. 
Do  we  make  all  five  survey  responses  point  decisively  to  communication 
as  a  problem  before  we  assert  that  the  hypothesis  claim  is  supported? 
Indeed,  this  requirement  seems  to  be  impractical.  Well  then,  how  many 
of  the  survey  responses  must  indicate  communication  as  a  problem  before 
the  hypothesis  claim  will  be  supported?  This  section  describes  a 
technique  which  will  be  used  to  objectively  evaluate  whether  or  not  the 
survey  responses  support  the  hypothesis  claims.  First,  the  information 
available  for  determining  an  objective  answer  to  the  focus  question  for 
each  hypothesis  will  be  listed.  Then,  the  evaluation  technique  will  be 
presented  and  explained.  Finally,  a  table  will  be  developed  which  will 
be  filled  in  with  Chapter  IV  results  to  determine  what  the  answer  is  to 
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each  of  the  four  focus  questions.  These  answers  will  be  used  to 
indicate  support  for  the  four  major  hypotheses. 

The  Available  Information.  The  method  described  thus  far 
will  provide  the  following  information:  1)  what  value  should  the  mean 
response  be  to  indicate  a  "no"  answer  to  the  focus  question  being 
considered,  2)  the  sample  mean  of  each  of  the  12  survey  responses,  3) 
the  value  of  the  Low  Group's  mean,  4)  the  value  of  the  High  Group's 
mean,  5)  a  confidence  interval  about  the  mean  response  which  will  show 
whether  or  not  the  population  mean  is  definitely  to  the  "correct"  side 
of  the  value  ot  three,  and  6)  an  indication  of  whether  or  not  the  two 
High  and  Low  Group  means  are  equivalent.  This  information  will  be  used 
to  develop  an  objective  test  that  will  be  used  to  determine  whether  or 
not  the  survey  responses  support  the  hypothesis  claims. 

Evaluation  Technique.  The  researcher  will  assess  a  "value" 
to  data  items  2  through  6  above  if  those  data  items  indicate  support  for 
the  hypothesis  claim.  The  values  will  be  awarded  if  the  average  or  the 
interval  is  on  the  side  of  the  value  of  three  needed  to  support  the 
hypothesis  claim.  For  example,  for  the  first  focus  question,  the 
response  of  Survey  Question  Number  1  needs  to  be  less  than  three  to 
indicate  a  "no"  response  to  the  focus  question.  So,  if  the  average 
response  is  less  than  three,  then  the  survey  question  will  be  awarded  a 
score.  Likewise,  if  both  group  means  are  below  the  value  of  three,  then 
the  survey  question  will  be  awarded  a  higher  score.  The  point  values 
will  be  assessed  as  follows: 

0.15  points  if  the  mean  response  indicates  a  "no"  answer  to  the  FQ 
0.10  points  if  the  Low  Group’s  mean  indicates  a  "no" 

0.15  points  if  the  High  Group's  mean  indicates  a  "no” 
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0.45  points  if  the  entire  interval  about  the  mean  indicates  a  "no" 
0.15  points  if  both  group  means  support  the  mean  and  are 
determined  to  be  equivalent 

Thus,  the  maximum  score  that  each  survey  question  may  be  awarded  is  1.0 
points.  The  points  will  be  awarded  to  each  survey  question.  Then  the 
points  will  be  summed  for  each  focus  question.  The  sum  will  be  divided 
by  the  number  of  survey  questions  being  used  to  answer  the  focus 
question  so  that  a  ratio  will  be  calculated.  If  the  ratio  is  greater 
than  or  equal  to  0.6,  then  the  overall  answer  to  the  focus  question  will 
be  determined  to  be  "no,"  thereby  indicating  support  for  the  hypothesis 
claim  from  which  the  focus  question  was  generated.  So,  we  now  have  an 
objective  test  that  can  be  used  to  determine  whether  the  survey 
questions  tend  to  support  the  hypothesis  claims. 

Table .  Table  9  shows,  for  each  major  hypothesis  and  for 
each  survey  question,  how  the  net  results  of  the  computations  in  this 
section  will  be  recorded.  The  values  awarded  to  the  survey  questions 
will  be  included  on  the  table,  and  they  will  be  summed.  Each  major 
hypothesis'  ratio  will  be  computed  by  dividing  the  sum  of  the  survey 
question  total  scores  by  the  number  of  survey  questions  used  to  test  the 
hypothesis.  If  the  ratio  is  greater  than  0.6,  then  the  hypothesis  claim 
will  be  accepted.  If  the  ratio  is  less  than  or  equal  to  0.6,  then  the 
hypothesis  claim  will  be  rejected. 

Chapter  Summary 

In  summary,  this  chapter  provides  the  methodology  for  determining 
not  only  whether  or  not  the  ECP  process  is  untimely  with  respect  to  the 
time  guidelines  provided  in  the  CMP,  but  also  whether,  in  a  general 
sense,  the  SPO  personnel  involved  with  the  process  believe  that  those 
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TABLE  9 


THE  END  RESULTS  CF  THE  TESTS  OF  THE  HYPOTHESES 


MR  Needed 

to  Support  Points  Awarded  For  Max 

Claim  MR  LGM  HGM  Intvl  Iden  Score 


Survey  Question 

1 

MR 

<  3 

Values 

Up  to 

Survey  Question 

2 

MR 

<  3 

0.15  0. 

.10  0.15  0.45 

Survey  Question 

3 

MR 

<  3 

Survey  Question 

4 

MR 

<  3 

Survey  Question 

5 

MR 

>  3 

Hypothesis  I 

Ratio : 

(Sum 

of  SQs  1 

-  5)  /  5  - 

1.00 


1.00 


Survey  Question  6  MR  <  3 
Survey  Question  7  MR  <  3 
Survey  Question  8  MR  <  3 

Hypothesis  II  Ratio:  (Sum  of  SQs  6-8)  /  3  =  1.00 


Survey  Question  9  MR  <  3 
Survey  Question  10  MR  >  3 

Hypothesis  III  Ratio:  (Sum  of  SQs  9  &  10)  /  2  =  1.00 


Survey  Question  11  MR  <  3 
Survey  Question  12  MR  >  3 

Hypothesis  IV  Ratio:  (S  SQs  11  &  12)  /  2  =  1.00 


CMP  times  are  reasonable  and  that  problems  with  the  process  itself  are 
driving  the  timeliness  problem.  Chapter  IV  will  provide  the  results  and 
summaries  of  the  findings  obtained  from  performing  the  steps  in  this 
chapter,  and  Chapter  V  will  present  a  summary  of  the  thesis  effort,  the 
conclusions,  and  the  recommendations. 


Ill 


IV.  Results  and  Findings 


Chapter  III  presented  a  "cookbook"  of  steps  to  be  performed  and  it 
stated  that  the  results  would  be  presented  in  Chapter  IV.  Thus  it  is  in 
this  chapter  that  the  reader  will  begin  to  see  whether  or  not  the  ACM 
SPO  ECP  process  is  untimely,  with  respect  to  the  times  stated  within  the 
CMP,  and  whether  or  not  the  hypotheses,  presented  in  Chapter  I,  are 
valid  claims.  First,  the  results  of  the  study  of  ACM  SPO  ECP  process' 
timeliness  will  be  presented.  Then,  the  results  from  the  analysis  of 
the  survey  questionnaires  will  be  summarized  and  statements  regarding 
the  hypothesis  claims  will  be  made. 

Timeliness  of  ECP  Process 

In  this  section,  the  results  of  the  data  analysis  that  was  used  to 
answer  the  five  focus  questions  will  be  presented. 

FO  Number  1.  Is  the  ACM  SPO  ECP  process  taking  too  much  time? 
Forty-three  ECPs  met  the  criteria  for  selection. 

Results.  The  average  number  of  days  the  ECPs  took  to  be 
processed  was  220.2  days  with  a  sample  standard  deviation  of  80.6  days. 
The  LBV  was  calculated,  and  Figure  12  shows  the  results.  The  left 
vertical  bar  indicates  the  CMP  standard  of  147  days.  The  right  bar 
shows  where  the  sample  mean  and  LBV  fell  with  respect  to  the  standard; 
namely,  the  LBV  is  greater  than  the  standard  value  of  147  days. 

Findings .  Since  the  LBV  is  greater  than  the  CMP  standard, 
the  answer  to  the  focus  question  is  "yes,  the  process  is  indicated  as 
untimely."  With  this  answer,  attempts  to  answer  the  next  four  focus 
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Figure  12.  Results  for  Focus  Question  Number  1 


questions  may  pinpoint  which  specific  areas  of  the  ECP  process  are 
having  the  most  difficulty  with  timeliness. 

FQ  Number  2.  Are  individual  process  steps  taking  too  much  time? 

Results .  Table  10  below  provides  a  summary  of  the  results 
collected  by  performing  the  methodology  for  FQ  Number  2.  For  each 
different  process  step,  the  table  provides  the  following  information: 
the  identified  shape  of  the  distribution  (for  sample  sizes  greater  than 
30,  "CLT"  is  listed  to  signify  the  use  of  the  Central  Limit  Theorem), 
the  number  of  ECPs  that  travelled  through  that  particular  step,  the 
average  number  of  days  each  ECP  took  to  be  processed  within  that  step, 
the  sample's  standard  deviation,  the  LBV,  and  the  goal  provided  by  the 
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TABLE  10 


RESULTS  OF  SPREADSHEET  MANIPULATION 


#  of 

Sample 

Occur- 

Sample 

Std 

Goal 

Step  From  / 

To 

ShaD 

e  rences 

Mean 

Dev 

n 

LBV 

(CMP) 

TCP 

1 

-  ECP 

1 

E 

5 

40.6 

21 

8 

25 

.4 

52 

TCP 

1 

-  TCM 

1 

CLT 

49 

23.1 

13 

8 

20 

.6 

14 

TCM 

1 

-  ECP 

1 

CLT 

45 

70.2 

48 

4 

61 

.9 

38 

TCM 

1 

-  TCP 

2 

E 

4 

47.7 

25 

7 

28 

.6 

TCP 

2 

-  TCM 

2 

E 

5 

27.0 

29 

7 

16 

.9 

TCM 

2 

-  ECP 

1 

E 

5 

51.2 

39 

4 

32 

.0 

ECP 

1 

-  CCB 

1 

CLT 

60 

30.2 

27 

7 

25 

.6 

28 

CCB 

1 

-  ECP 

2 

CLT 

47 

37.2 

33 

5 

30 

.9 

17 

CCB 

1 

-  REV 

E 

5 

54.8 

47 

8 

34 

.  3 

57 

CCB 

1 

-  COM 

OUT 

N 

8 

33.5 

25 

8 

20 

.  6 

7 

ECP 

2 

-  OPR 

1 

CLT 

39 

19.6 

17 

4 

16 

.0 

ECP 

2 

-  CCB 

2 

N 

8 

12.6 

9 

2 

8 

.0 

CCB 

2 

-  ECP 

3 

E 

7 

22.4 

20 

2 

14 

.  9 

ECP 

3 

-  OPR 

1 

N 

5 

2.4 

2 

6 

0 

.6 

OPR 

1 

-  REV 

CLT 

34 

38.6 

32 

5 

31 

.4 

OPR 

1 

-  OPR 

2 

N 

10 

51.2 

35 

5 

35 

.7 

OPR 

2 

-  REV 

N 

10 

33.8 

31. 

8 

19 

.9 

REV 

-  S/A 

CLT 

51 

9.4 

16 

8 

6 

.4 

101 

CMP. 

No 

times 

are 

listed 

under  the 

"Goal"  column  when 

the  CMP 

does  not 

address  a  time  requirement.  For  those  process  steps  that  had  more  than 
10  occurrences,  Figure  13  provides  a  pictorial  view  of  the  relationship 
between  the  CMP  stated  times,  when  those  times  are  provided  in  the  CMP, 
and  the  lower  bound  values.  The  CMP  times  are  indicated  by  virtue  of 
the  height  of  the  thinner  and  more  darkly  shaded  bars  adjoining  the  left 
edges  of  the  data  bars.  By  simply  looking  at  Figure  13,  and  by  studying 
Table  10,  one  can  determine  which  process  steps  are  indicated  as 
possibly  being  untimely  by  comparing  the  LBV,  represented  by  the  lower 
edge  of  the  shaded  region  of  the  data  bars,  to  the  stated  amount  of  time 
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MOST  FREQUENTLY  ACCESSED  STEPS 


PROCESS  STEP 


Figure  13.  Results  from  FQ  Number  2  Data  Analysis 


the  process  should  take.  These  potentially  untimely  steps  are  listed 
below. 


TCP  Receipt-TCM: 

LBV  =  20.5  days  versus 

CMP  goal  of  14  days 

TCM-ECP  Receipt: 

LBV  -  61.0 

ft  ii  ii  It 

CCB-GD/C  Response: 

LBV  =31.2 

17 

Resp-Rev  Complete: 

LBV  =  16.1+32.5 

40 

OPR-OPR-Rev  Complete: 

MTs  =  51.2+33.8 

0 

Some  process  steps  were  indicated  as  possibly  being  more  timely  than  the 
times  presented  in  the  CMP.  For  example,  ECPs  that  were  approved  at  the 
CCB  were  allowed  57  days  to  have  all  contractual  review  steps  completed. 
However,  these  ECPs  took  on  average  less  than  55  days,  and  the 
population  mean  could  be  as  low  as  only  35  days.  Also,  the  ECP  review 
step  had  a  lower  bound  value  of  25.6  days,  which  is  less  than  the  CMP 
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goal  of  28  days,  and  the  supplemental  agreement  step  took  on  average  9.4 
days,  less  than  the  goal  of  10  days.  The  process  steps  for  which  the 
CMP  did  not  provide  times  proved  to  add  significant  time  to  the  total 
process.  For  example,  it  took  approximately  126  days  on  average  to 
process  ECPs  through  the  second  TCM,  when  a  second  TCM  was  required.  It 
took  another  51.2  days,  on  average,  to  process  ECPs  through  a  second  OPR 
Approval  step. 

Findings .  The  findings  here  are  that  certain  process  steps 
appear  to  be  more  untimely  than  others .  While  the  ACM  SPO  seems  to  be 
doing  a  fair  job  of  boarding  an  ECP  at  the  CCB  within  the  time  outlined 
in  the  CMP,  it  is  taking  approximately  one  week  too  long  to  conduct  a 
TCM  and  about  two  weeks  longer  than  it  should  take  to  submit  CCBDs  to 
the  contractor  when  ECPs  are  disapproved  (the  CCB  1  -  COMments  OUT 
step).  Furthermore,  the  time  it  took  for  the  SPO  to  receive  the  ECP 
after  the  TCM  was  conducted  was,  on  average,  70.2  days.  The  CMP 
provides  only  38  days  for  this  step.  The  initial  tendency  is  to  perhaps 
blame  the  contractor.  However,  within  the  70  days,  the  contractor  has 
been  preparing  TCM  minutes  that  the  SPO  has  to  approve  before  the 
contractor  can  begin  preparing  the  ECP.  Also,  for  proposals  that  are 
originally  submitted  with  a  cost  volume,  the  contractor  must  take  time 
to  prepare  the  cost  volume  after  the  TCM.  So,  further  research  is 
needed  in  this  area  to  determine  where  the  bulk  of  the  time  delay  is  and 
to  determine  if  the  CMP  perhaps  does  not  provide  enough  time  to  perform 
this  process  step. 

FQ  Number  3.  Are  the  ACM  SPO  ECPs  being  routed  through  the  same 
process  steps  more  times  than  once? 
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Results .  The  results  of  the  data  gathering  effort  are 
indicated  in  Figure  14  and  Figure  15.  Sixty  proposals  had  activity 
during  the  seven  month  period  and  were  subsequently  closed  by  August  7, 
1990.  Figure  14  shows  the  routes,  and  the  number  of  times  the  route  was 
accessed,  from  TCP  receipt  to  the  CCB .  Five  proposals  entered  the 


Figure  14.  Process  Flows  from  TCP  to  CCB 


system  without  TCPs  or  TCM,  while  the  remaining  55  entered  the  system  in 
the  form  of  a  TCP.  All  60  made  it  to  a  CCB,  but  five  proposed  changes 
required  an  updated  TCP  before  the  ECP  was  submitted.  The  boid  lines 
and  boxes  indicate,  for  Figure  14,  not  only  the  ideal  flow  but  also  the 
flow  most  frequently  used  by  these  60  ECPs .  In  fact,  45  of  the  60  ECPs , 
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or  75%,  followed  the  ideal  flow  up  to  the  CCB.  However,  this  percentage 
drops  dramatically  for  the  steps  beyond  the  CCB,  as  Figure  15  indicates. 
To  refresh  the  reader's  memory,  the  ideal  flow  would  include  approval  of 


Fig1  re  15.  Process  Flows  from  CCB  to  S/A 


the  ECP  at  the  CCB  followed  immediately  by  the  review  process  and 
supplemental  agreement.  However,  the  ^ost  frequent  disposition  of  the 
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ECPs  at  the  CCB  was  "approved  with  comment,"  which  meant  that  these  ECPs 
needed  to  be  updated  with  the  CCB  comments  before  they  could  be  approved 
and  before  the  review  process  could  begin.  The  frequency  of  ECPs 
getting  approved  with  comments  was  47  times  out  of  60,  or  roughly  78%, 
while  the  frequency  at  which  the  ECPs  were  approved  as  written,  the 
ideal  disposition,  was  only  five  times  out  of  the  60,  or  about  8%.  In 
other  words,  the  proportion  of  ECPs  that  are  Approved  with  Comments, 
defined  as  pa  in  Chapter  III,  is  0.78.  This  proportion  is  much  higher 
than  that  of  0.30  which  was  proposed  by  the  researcher  as  the  test  to 
determine  whether  process  steps  were  having  to  be  re- accomplished . 
Furthermore,  of  the  ECPs  that  were  eventually  approved,  47  of  52,  or 
90.4%,  had  to  have  processing  steps  re-accomplished.  The  other  test 
provided  in  Chapter  III  was  to  compare  the  percentage  of  the  number  of 
ECPs  that  did  not  get  approved  by  the  OPR  at  the  first  OPR  Approval 
decision  point  to  the  value  of  0.20.  Figure  15  shows  that  of  44 
proposals  arriving  at  the  first  OPR  Approval  decision  point,  10  of  them, 
or  22.7%,  were  not  approved  by  the  OPR.  Other  information  can  be  pulled 
from  Figure  15  also.  For  example,  eight  of  the  47,  or  17%,  of  the  ECPs 
that  were  boarded  at  the  ECP  required  changes  sufficient  enough  that  the 
ECP  had  to  be  brought  back  to  the  CCB  for  approval.  Furthermore,  of 
those  eight  ECPs,  seven  required  even  further  updates  before  they  could 
be  approved. 

Findings.  The  major  finding  is  that  the  ECPs  are  having  to 
have  process  steps  re  -  accomplished .  This  may  be  necessary  because  of 
inadequate  communication  between  the  SPO  and  the  contractor,  inadequate 
TCP  review  and  TCM  process  steps  on  the  part  of  the  SPO,  or  an  inability 
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on  the  part  of  the  contractor  to  follow  the  directions  outlined  in  the 
TCM  minutes  or  in  the  CCBD.  The  analysis  of  the  survey  will  possibly 
reveal  one  or  more  of  these  potential  problems  as  the  reason  why  ECPs  do 
not  follow  the  ideal  process  flow,  but  instead  must  be  processed  through 
certain  steps  more  than  once. 

FO  Number  4.  Are  there  any  specific  ECP  process  flow  paths  that 
are  more  timely  than  the  CMP  requirements? 

Results .  Recall  that  Table  6  in  Chapter  III  (see  page  84) 
presented  a  list  of  the  nine  different  process  flow  times  and  the  number 
of  days  the  CMP  stated  that  the  flows  should  take.  The  Table  6  data  is 
presented  here  in  Table  11,  which  provides  not  only  the  Table  6  informa¬ 
tion  but  also  the  actual  flow  averages  and  LBVs.  Figure  16  presents  the 

TABLE  11 


PROCESS  FLOW  PATHS,  CMP  TIMES,  AND  ACTUALS  (9:3-2) 


Days 

Avg 

Path  Number  and  Process  Flow 

CMP 

Davs 

LBV 

1 . 

TCP-TCM-ECP-CCB-ECP-OPR-REV-S/A 

147 

233.1 

209.5 

2. 

TCP  - TCM - ECP - CCB - ECP - OPR - OPR - REV -  S/A 

147 

251.0 

209.8 

3. 

TCP-ECP-CCB-ECP-OPR-REV-S/A 

147 

252.8 

241.2 

4. 

TCP -TCM -ECP -CCB -REV -S/A 

147 

107.2 

90.1 

5. 

TCP -TCM -ECP -CCB -COMM  OUT 

97 

183.8 

148.5 

6 . 

TCP - TCM - TCP -TCM -ECP- CCB -- -COMPLETION 

147 

238.5 

205.1 

7. 

ECP-CCB- - -S/A 

95 

159.7 

53.8 

8. 

MISCELLANEOUS 

126 

205.7 

193.8 

9  . 

ENTRY - COMPLETION 

138 

215.9 

209.2 

Note 

i:  time  allowances  assume  the  ECP  is 

a  no-cost 

change . 

same  information  in  the  form  of  a  bar  chart.  The  heights  of  the  more 
darkly  shaded  and  thinner  bars,  called  "standard"  bars,  provide  a 
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^ure  16.  Average  Path  Flow  Times 


picture  of  how  much  time  is  provided  for  each  path  in  the  CMP,  and  were 
determined  based  on  the  CMP  time  allowances  for  completing  the  major 
process  steps  in  the  flow  path.  For  example ,  Path  8's  and  Path  9's 
standard  bars'  heights  were  calculated  by  averaging  the  time  allowed  for 
each  ECP  based  on  each  ECP's  path  flow.  The  bottom  edge  of  the  shaded 
region  on  the  data  bars  indicates  the  lower  bound  value.  Thus,  by 
comparing  the  tops  of  the  standard  bars  to  the  bottoms  of  the  shaded 
regions  on  the  data  bars,  one  can  see  whether  or  not  the  process  flow 
path  provided  approved  ECPs  faster  than  the  time  requirements  spelled 
out  in  the  CMP.  Only  two  paths  potentially  qualify  for  this 
distinction,  Path  4  and  Path  7.  Path  7  ECPs  did  not  go  through  the  TCP 
Review  and  TCM  process  steps  because  a  TCP  was  never  submitted.  On  the 
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surface,  one  could  say  that  ECP  process  timeliness  will  improve  by 
eliminating  these  steps;  however,  only  three  ECPs  are  included  in  this 
sample,  and  the  variation  in  processing  time  between  them  was  relatively 
high,  explaining  why  the  interval  between  the  sample  mean  and  the  lower 
bound  value  appears  to  be  so  large.  Furthermore,  the  average  of  the 
three  Path  7  ECPs  was  almost  160  days,  168%  more  than  that  allowed  in 
the  CMP.  The  other  potentially  timely  path  is  Path  4.  This  path  is  the 
ideal  path  flow  for  ECPs  to  follow,  and,  since  it  is  indicated  as 
potentially  timely,  provides  some  evidence  that  if  the  process  steps  are 
done  correctly  the  first  time,  a  timely  process  can  result. 

Findings .  ECPs  that  follow  Path  4  and  Path  7  were  timely 
enough  to  allow  them  to  be  candidates  for  timely  ECP  path  flows.  Of  the 
two,  Path  7  appears  to  qualify  for  the  distinction  only  because  the 
small  sample  size  and  the  relatively  wide  variation  in  the  data  forced 
the  lower  bound  value  to  be  below  the  time  requirement.  Path  4,  on  the 
other  hand,  had  a  relatively  small  variation  in  the  data,  plus  the 
sample  mean  was  relatively  far  below  the  time  requirement.  The  finding 
here  is  that  yes,  one  process  flow  path  appears  to  be  more  timely  than 
the  CMP  requirement;  indeed,  that  path  was  the  only  path  that  did  not 
have  ECPs  re-accomplishing  process  steps.  Thus,  a  correlation  seems  to 
exist  between  timeliness  and  "doing  things  correct  the  first  time,"  and 
the  indication  is  that  the  ACM  SPO  ECP  process  may  be  able  to  become  a 
timely  process  if  efforts  are  taken  to  reduce  the  frequency  of  re- 
accomplishing  process  steps. 


122 


FQ  Number  5 .  Is  the  number  of  open  ECPs  awaiting  processing  in 
any  particular  process  step  disproportionate  to  the  number  that  should 
be  waiting  in  that  step? 

Results .  Sixty- four  ECPs  were  open  on  August  3,  1990. 
Table  12  below  shows,  for  each  of  the  six  process  steps,  the  expected 
number  of  ECPs  in  each  step,  the  actual  number  in  the  step,  the  step's 


TABLE  12 

COMPUTATIONS  AND  RESULTS  OF  TEST  OF  PROPORTIONS 


Actual 

Expected 

Expected 

Contribn 

Count 

Percent 

Number 

to 

Waiting  for 

Ni 

pi 

npi 

v2i 

TCM  Approval 

5 

9.5 

6.08 

0.192 

ECP  #  1 

10 

25.9 

16.58 

2.611 

CCB  #  1 

12 

19.0 

12.16 

0.002 

CD/C  RESP 

7 

11.6 

7.42 

0.024 

Review 

27 

27.2 

17.41 

5.282 

S/A 

3 

6.8 

4.35 

0.419 

Total 

64 

100.0 

64.00 

Total  Computed  y2 

Test  Statistic  Value 

8.530 

The  y2  Critical  Value  (11: 

636) 

9.236 

Note:  20  of  the  27  waiting  for  Review  are  actually  waiting  for 

OPR  Approval 


contribution  to  the  chi-squared  test  statistic,  and  the  total  chi- 
squared  test  statistic  value.  By  comparing  the  calculated  test 
statistic  value  to  the  chi-squared  critical  value,  it  is  apparent  that 
the  data  does  not  indicate  a  disproportionate  distribution  of  the  ECPs 
in  the  process  steps.  The  largest  contribution  to  the  calculated  test 
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statistic  value  was  from  the  Contractual  Review  process  step.  An 
anomaly  of  the  data  is  that  20  of  the  27  ECPs  in  that  category  were 
actually  waiting  for  the  OPR  to  approve  the  contractor's  latest  ECP 
submission. 

Findings .  The  null  hypothesis,  which  stated  that  the  actual 
proportions  equal  the  expected  proportions,  could  not  be  rejected.  The 
data  does  not  indicate  a  problem  with  the  distribution  of  open  ECPs  in 
the  ECP  process  steps.  However,  the  fact  that  20  of  27  ECPs  recorded  in 
the  contractual  review  category  were  waiting  for  OPR  approval  indicates 
a  potential  problem.  Perhaps  OPRs  are  being  overworked  and  are  unable 
to  devote  time  to  these  ECPs  that  require  their  attention.  Perhaps  too 
the  ECP  reports  from  which  the  data  was  taken  were  not  updated.  In  any 
case,  the  incident  is  cause  for  some  concern  and  perhaps  some  additional 
study. 

Summary  of  Results  to  Determine  Process  Timeliness.  Briefly,  in 
this  section  five  focus  questions  were  answered.  Four  of  the  focus 
questions  were  answered  with  a  "yes,"  tending  to  indicate  support  for 
the  claim  that  the  ECP  process  is  not  timely,  at  least  with  respect  to 
the  CMP's  stated  times.  Next,  we  attempt  to  determine  whether  the 
process  is  untimely  or  if  the  CMP's  processing  times  are  unrealistic. 

Analyze  the  Survey  Questionnaire  Results 

General  Results.  Table  13  below  presents  the  results  of  the  data 
analysis,  spelling  out  the  survey  question  number,  the  value  (MR)  of  the 
mean  response  needed  to  support  the  hypothesis  claim  (whether  less  than 
or  more  than  the  value  of  three),  the  Low  Group,  High  Group,  and  Total 
mean  responses,  whether  or  not  the  Low  and  High  Group  means  are 
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TABLE  13 


INFORMATION  OBTAINED  FROM  METHODOLOGY 


MR  Needed 

to  Support 

Means 

Response 

- 

Claim 

LGM/HGM/Tot 

Consensus? 

Definite? 

Focus  Question 

I 

SQ  1 

MR 

< 

3 

3.28/  3.78/ 

3.51 

Yes 

No 

UBV-3 . 69 

SQ  2 

MR 

< 

3 

1.88/  3.46/ 

2.66 

No 

Yes 

UBV=2 . 85 

SQ  3 

MR 

< 

3.67/  4.57/  4.01 

No 

No 

UBV=4 . 15 

SQ  4 

MR 

< 

3 

3.11/  4.50/ 

3.55 

No 

No 

UBV=3 . 82 

SQ  5 

MR 

3 

3.22/  3.63/ 

3.36 

Yes 

Yes 

LBV =3 . 13 

Focus  Question 

II 

SQ  6 

MR 

< 

3 

2.18/  3.25/ 

2.72 

No 

Yes 

UBV=2 . 86 

SQ  7 

MR 

< 

3 

2.94/  4.08/ 

3.46 

No 

No 

UBV=3 . 60 

SQ  8 

MR 

< 

3 

3.82/  4.28/ 

3.53 

No 

No 

UBV-3.69 

Focus  Question 

III 

SQ  9 

MR 

< 

3 

2.85/  4.04/ 

3.33 

No 

No 

UBV=3 .47 

SQ  10 

MR 

> 

3 

3.50/  3.88/ 

3.60 

Yes 

Yes 

LBV=3 . 3  7 

Focus  Question 

IV 

sq  ll 

MR 

< 

3 

2.52/  3.00/ 

2.77 

Yes 

Yes 

UBV-2.99 

SQ  12 

MR 

> 

3 

3.22/  3.86/ 

3.38 

Yes 

Yes 

LBV=3 . 17 

statistically  close  (consensu?',  and  whether  or  not  the  90%  confidence 
interval  about  the  total  mean  indicates  a  definite  response  that 
supports  the  hypothesis  claim  (the  yes  or  no  answer  is  provided  along 
with  the  value  of  either  the  upper  bound  value  or  the  lower  bound 
value).  As  a  note  of  explanation,  if  the  answer  to  "Consensus"  is  yes, 
then  that  means  that  the  High  Group  and  the  Low  Group  means  are 
statistically  close.  It  does  not  mean  that  the  data  indicates  support 
for  the  hypothesis  claim.  The  consensus  information  is  provided  so  that 
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the  reader  may  understand  which  questions  produced  the  same  general 
feeling  in  the  people  throughout  those  that  answered  the  survey. 

Figure  17  provides  a  visualization  of  the  consensus  in  the  SPO  about  the 
twelve  different  survey  questions.  It  shows  the  Low  Group's  mean,  the 
mean  of  all  of  the  responses,  and  the  High  Group's  mean,  and  it  shows 
the  relative  variation  between  the  two  group's  mean  responses.  For 


Figure  17.  A  Picture  of  Consensus 


example,  Survey  Question  Number  6  produced  a  marked  difference  in  group 
mean  scores,  while  Survey  Question  Number  12  produced  fairly  consistent 
mean  responses.  Figure  18,  on  the  other  hand,  shows  whether  or  not  the 
population  mean  lies  to  the  "correct"  side  of  the  value  of  three  to 
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90%  ONE-SIDED  CONFIDENCE  INTERVALS 


Figure  18.  90%  Confidence  Intervals  About  the  Means 


support  the  hypothesis  claims.  For  Survey  Question  Number  2,  the  shaded 
region  represents  the  entire  interval  within  which  the  population  mean 
response  lies,  with  90%  confidence.  Since  the  entire  interval  is  below 
the  value  of  three,  support  for  the  hypothesis  claim  is  indicated. 

Also,  for  Survey  Question  Number  10,  the  shaded  region  extends  up  from 
the  lower  bound  value  of  3.37  to  3,  the  highest  possible  response. 

Thus,  support  for  the  hypothesis  claim  is  shown.  However,  the  interval 
for  Survey  Question  Number  7  extends  from  the  minimum  possible  value  of 
1.0  up  to  3.6.  Since  the  interval  includes  the  value  of  three,  support 
for  the  hypothesis  claim  is  not  indicated.  So,  Figure  17  and  Figure  18 
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provide  snapshots  of  the  data  now  available  to  conduct  the  major  tests 
of  the  hypothesis  claims. 

General  Findings.  A  discussion  of  general  findings  will  be 
reserved  until  after  the  test  of  the  hypothesis  claims  is  conducted. 

The  reason  for  the  delay  is  because  the  researcher  believes  that  the 
data  needs  to  be  taken  as  a  whole,  rather  than  discussing  each  bit  of 
information  separately.  For  this  reason  the  hypothesis  test,  discussed 
below,  was  developed. 

The  Hypotheses  Tested.  Table  14  shows  the  results  of  filling  in 
Chapter  Ill's  Table  9  information.  The  results  indicate  that  the  test 
ratio,  determined  by  dividing  the  sum  of  the  chi-squared  contributions 
by  the  number  of  survey  questions  used  to  test  the  hypothesis,  is  less 
than  0.5  for  both  the  Communication  hypothesis  and  the  ECP  Submissions 
hypothesis.  However,  the  test  ratio  is  greater  than  0.5  for  both  the 
Personnel  Resources  and  TCM  Preparation  hypotheses.  Findings  from  these 
results  will  now  be  summarized. 

Findings .  Although  the  selection  of  the  critical  ratio  was 
arbitrary,  at  least  now  the  major  problem  areas  in  the  ACM  SPO  ECP 
process  have  been  highlighted.  Namely,  people  tend  to  feel  as  though 
they  lack  the  time  to  review  ECPs  and  provide  comments  by  the  date 
requested.  Furthermore,  OPRs  feel  as  though  they  need  to  go  to  the 
reviewers  to  ask  for  ECP  comments,  indicating  that  they  believe  the 
reviewers  are  not  responsive  to  the  suspense,  for  whatever  reason. 
Additionally,  strong  support  exists  for  the  assertion  that  people  are 
not  taking  the  time  to  review  TCPs  and  attend  TCMs .  Indeed,  by  focusing 
more  on  this  area,  perhaps  a  vast  improvement  could  be  made  to  the 
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TABLE  14 


THE  END  RESULTS  OF  THE  TESTS  OF  THE  HYPOTHESES 


MR  Needed 

to  Support 

Points  Awarded  For 

Max 

Claim  MR 

LCM 

HGM 

Intvl 

Iden 

Score 

Survey  Question 

1 

MR  <  3  0 

0 

0 

0 

0 

0 

Survey  Question 

2 

MR  <  3  0.15 

0.10 

0 

0.45 

0 

0.70 

Survey  Question 

3 

MR  <  3  0 

0 

0 

0 

0 

0 

Survey  Question 

4 

MR  <  3  0 

0 

0 

0 

0 

0 

Survey  Question 

5 

MR  >  3  0.15 

0.10 

0.15 

0.45 

0.15 

1.00 

Hypothesis  I 

Ratio:  (Sum  of  SQs  1  - 

5)  /  5 

= 

0.34 

Survey  Question 

6 

MR  <  3  0.15 

0.10 

0 

0.45 

0 

0.70 

Survey  Question 

7 

MR  <  3  0 

0.10 

0 

0 

0 

0.10 

Survey  Question 

8 

MR  <  3  0 

0.10 

0 

0 

0 

0.10 

Hypothesis  II  Ratio:  (Sum  of  SQs  6 

8)  / 

3  - 

0.30 

Survey  Question 

9 

MR  <  3  0 

0.10 

0 

0 

0 

0.10 

Survey  Question 

10 

MR  >  3  0.15 

0.10 

0.15 

0.45 

0.15 

1.00 

Hypothesis  III 

Ratio:  (Sum  of 

SQs  9 

&  10) 

/  2  - 

0.55 

Survey  Question 

11 

MR  <  3  0.15 

0.10 

0.15 

0 

0.15 

1.00 

Survey  Question 

12 

MR  >  3  0.15 

0.10 

0.15 

0.45 

0.15 

1.00 

Hypothesis  IV  Ratio:  (Sum  of  SQs  11 

&  12) 

/  2  = 

1.00 

timeliness  of  the  ECP  process  by  virtue  of  a  reduction  in  the  frequency 
at  which  ECPs  must  be  processed  through  the  OPR  Approval  process  step, 
for  example. 

Chapter  Summary 

In  this  chapter  the  results  obtained  from  performing  the  steps 
outlined  in  Chapter  III  were  presented.  Briefly,  the  researcher 
determined  that  four  of  the  ECP  timeliness  focus  questions  had  yes 
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answers.  Namely,  the  overall  process  was  found  to  be  taking  too  much 
time;  specific  process  steps  were  found  to  be  more  untimely  than  others, 
including  the  process  steps  from  TCP  Receipt  to  TCM,  TCM  to  ECP  Receipt, 
CCB  to  GD/C  Response,  Response  to  Contractual  Review  Complete,  and  OPR 
Approval  to  OPR  Approval  to  Contractual  Review  Complete;  ECPs  are  being 
routed  through  process  steps  more  than  once;  and  the  ideal  path  flow's 
average  processing  time  was  the  only  process  flow  path  that  clearly  took 
less  time  on  average  than  the  times  stated  in  the  CMP.  However,  the 
answer  to  the  fifth  timeliness  focus  question  was  no,  indicating  that 
the  open  ECPs  were  not  distributed  disproportionately.  Furthermore,  the 
survey  responses  indicated  a  lack  of  personnel  resources  (in  a  generic 
sense)  and  a  lack  of  TCM  preparation  and  attendance  as  barriers  to 
process  timeliness,  while  communication  and  ECP  documentation  were  not 
indicated  as  problem  areas.  With  these  results  and  findings,  the 
researcher  will  now  provide  a  general  summary  of  the  thesis  effort, 
state  the  major  findings,  and  conclude  this  report  with  recommended 
courses  of  action. 
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V.  Conclusions  and  Recommendations 


Conclusions 

In  summary,  the  ECP  process  in  the  ACM  SPO  has  been  found  to  be 
untimely  with  respect  to  the  Configuration  Management  Plan.  Furthermore, 
the  overall  untimeliness  is  related  to  one  major  factor. 

The  Major  Factor.  The  major  factor  to  the  SPO's  untimely  ECP 
processing  is  the  lack  of  effort  applied  before  and  during  the  TCM,  as 
the  following  discussion  explains. 

Symptoms  of  Problems.  First  of  all,  specific  process  steps 
are  taking  too  long  to  accomplish.  These  process  steps  include  the  TCP 
review  effort  in  the  SPO  (three  weeks  instead  of  two) ,  the  ECP  prepara¬ 
tion  effort  that  takes  place  after  the  TCM  (nine  weeks  instead  of  five 
and  one-half  weeks),  and  the  steps  between  the  CCB  and  receipt  of  the 
contractor's  response  (31  days  instead  of  17  days).  This  untimeliness 
is  a  problem  with  respect  to  the  ability  of  the  SPO  to  process  ECPs  in 
an  overall  manner  of  timeliness;  however,  the  untimeliness  of  these 
steps  is  a  symptom  of  the  real  problem.  Another  symptom  of  the  real 
problem  is  the  frequency  which  certain  process  steps  must  be  re -accom¬ 
plished,  including  the  edit,  submittal,  and  approval  steps.  Indeed, 
each  of  these  steps  must  be  accomplished  at  least  twice  for  each  ECP 
that  is  Approved  with  Comments  at  the  CCB;  recall  that  90.4%  of  the  ECPs 
that  eventually  receive  government  approval  are  Approved  with  Comments 
the  first  time  they  are  presented  to  the  CCB.  This  level  of  rework  also 
could  be  causing  people  to  feel  as  though  they  do  not  have  enough  time 
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to  effectively  review  the  ECPs  for  which  they  are  responsible  for 
reviewing- -  the  third  symptom  of  the  real  problem. 

The  Real  Problem.  Just  what  is  the  real  problem?  The  real 
problem  is  that  the  ACM  SPO  personnel  are  not  devoting  adequate  up  front 
effort  to  the  TCP  Review  and  TCM  process  steps.  This  problem  impacts 
the  ability  of  the  SPO  to  approve  ECPs  the  first  time  they  are  presented 
at  the  CCB.  Furthermore,  due  to  the  requirement  for  the  additional 
paperwork  needed  to  process  these  Approved  with  Comment  ECPs,  the  people 
feel  overworked  and.  possibly,  frustrated.  Also,  by  aggressively 
reviewing  TCPs ,  the  frequency  of  Approved  ECPs  could  increase,  resulting 
in  the  elimination  of  the  57  days  that  the  process  takes  between  the  CCB 
and  OPR  Approval  process  steps.  Then,  the  workload  would  diminish  and 
SPO  personnel  would  then  perhaps  have  more  time  to  devote  to  better  and 
better  reviews  of  the  documentation.  However,  the  ACM  SPO  must  realize 
the  impact  an  inadequate  TCP  review  process  has  on  the  SPO,  which  leads 
to  the  first  recommendation. 

Recommendations 

Education .  The  first  recommendation  is  to  educate  the  SPO 
personnel  regarding  the  necessity  for  an  adequate  TCP  review.  By 
spending  the  time  to  review  the  TCP  for  not  only  content  but  also 
typographical  errors,  the  SPO  may  be  able  to  cut  down  the  ECP  processing 
time  by  simply  improving  the  percentage  of  ECPs  that  are  Approved,  as 
written.  Doing  this  will  reduce  the  frequency  of  ECPs  that  need  OPR 
Approval  after  the  CCB.  Recall  that  the  only  other  process  flow  that 
approached  the  timeliness  required  by  the  CMP  was  the  one  that  did  not 
include  TCP  Review  and  TCM  process  steps.  Indeed,  if  the  people  in  the 
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SPO  do  not  take  the  time  to  really  study  the  TCP,  but  instead  adopt  the 
philosophy  that  they  can  catch  any  errors  in  content  or  documentation 
during  the  ECP  review,  then  the  SPO  should  abolish  the  TCP  and  TCM 
concept  altogether.  Otherwise,  the  only  impact  these  steps  will  have  on 
*  the  timeliness  of  the  process  will  be  to  increase  it  by  the  93.3  days 

(the  CMP  calls  for  52  days)  that  the  SPO  has  been  averaging  between  TCP 
Receipt  and  ECP  Receipt.  Furthermore,  by  improving  the  frequency  of 
Approved  ECPs ,  the  amount  of  effort  required  to  process  the  changes  and 
revisions  of  the  Approved  with  Comments  ECPs  will  be  diminished.  Thus, 
people  may  then  begin  to  feel  as  though  they  have  more  time  to  ade¬ 
quately  review  the  documents  floating  across  their  desks.  Indeed,  the 
steps  that  are  required  to  re -accomplish  specific  process  steps  provide 
the  biggest  single  impact  to  process  timeliness:  it  took  on  average 
approximately  74  days  to  conduct  the  second  TCM  after  the  first  one  was 
held,  ECPs  that  were  Approved  with  Comments  took  another  57  days  between 
the  CCB  and  the  OPR  approval  steps,  and  ECPs  that  had  to  go  through  the 
OPR  Approval  step  a  second  time  took  an  extra  51  days  to  get  processed. 

Emphasis .  The  second  recommendation  is  for  management  to  place  a 
greater  emphasis  on  the  need  of  the  SPO  personnel  to  take  the  time  to 
adequately  review  TCPs .  This  greater  emphasis  could  be  accomplished 
simply  by  setting  up  a  program  to  reward  OPRs ,  and  reviewers,  for 
reviewing  TCPs  so  well  that  the  ECPs  can  be  approved  without  comments  at 
»  the  CCB.  In  fact,  even  an  extra  week  of  review  before  the  TCM  and  CCB, 

if  necessary,  would  be  in  order  especially  if  it  meant  one  less  ECP  that 
had  to  be  approved  with  comments.  Thus,  the  minimum  recommended  course 
of  action  is  for  the  SPO  to  begin  keeping  track  of  the  ability  of  the 
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OPRs  and  reviewers  to  present  flawless  ECPs  at  the  CCB.  An  additional 
step  is  to  set  up  an  awards  program  to  reward  those  personnel  who  have 
been  able  to  cut  down  the  number  "Approved  with  Comment"  ECPs.  However, 
this  awards  program  would  have  to  temper  the  OPRs  from  becoming 
advocates  of  the  proposal  simply  for  the  purpose  of  improving  their  and 
their  reviewers'  approval  track  records.  Of  course,  it  is  hoped  that 
the  reward  program  would  not  cause  this  kind  of  activity;  however,  if  it 
does,  it  could  still  produce  quality  decisions  if  the  CCB  players  adopt 
certain  roles.  The  chairman  would  retain  his  role  as  the  decision 
maker.  The  OPR  could  be  the  advocate  for  the  change,  and  the  board 
members,  either  specific  individuals  or  the  entire  group,  could  play  the 
role  of  antagonists  to  the  change.  Who  plays  the  different  roles  is  not 
that  important;  however,  that  someone  is  playing  opposing  roles  is 
important . 

Training .  The  third  recommendation  is  to  ensure  that  all 
personnel  who  review  TCPs  and  ECPs  understand  what  they  should  be 
looking  for,  and,  if  they  find  concerns,  how  they  should  attempt  to 
resolve  those  concerns.  Indeed,  any  actions  to  improve  the  ability  of 
the  SPO  personnel  to  perform  their  duties  is  warranted,  especially  in 
regards  to  the  ECP  process .  The  CPIT  has  recommended  a  training  program 
be  established;  this  researcher  does  not  disagree  with  the 
recommendation  of  that  team. 

Tracking .  Finally,  the  SPO  must  be  able  to  track  its  progress 
toward  imp  vement  in  the  timeliness  of  the  ECP  process.  The  methodol¬ 
ogy  presented  in  this  thesis  should  be  performed  once  every  six  months. 
The  result  would  be  detailed  information  which,  when  compared  to  the 
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results  tabulated  in  Chapter  IV,  would  indicate  whether  or  not  the  time¬ 
liness  of  the  SPO's  ECP  process  is  improving.  If  time  and  personnel 
constraints  limit  the  SPO  from  performing  all  of  the  steps  presented  in 
Chapter  III,  then  the  SPO  should  concentrate  first  on  the  frequency  that 
ECPs  are  approved  with  comments,  next  on  the  timeliness  of  individual 
process  steps,  then  on  the  simple  average  processing  time,  and  last  on 
the  individual  process  flow  path  times.  Also,  the  SPO  could  compute  and 
compare  only  the  average  times,  neglecting  the  variation  of  the  data; 
however,  the  statistical  accuracy  of  the  findings  would  be  reduced. 
Finally,  the  SPO  should  keep  asking  itself  whether  or  not  it  is  devoting 
time  to  the  TCP  Review  and  TCM  process  steps. 

Recommended  Future  Research 

Research  the  Contractor's  Process.  This  study  effort  was 
primarily  aimed  at  the  ability  of  the  ACM  SPO  to  process  its  ECPs. 
However,  some  of  the  findings  indicated  untimely  process  steps  that 
required  major  activity  on  the  part  of  the  contractor.  A  study  should 
be  conducted  to  determine  the  proper  number  of  personnel  the  contractor 
needs  to  prepare  TCPs,  ECPs,  CCPs ,  etc.,  and  cost  volumes  for  ECPs  and 
CCPs.  The  major  hypothesis  for  this  study  should  be  that  the  contractor 
does  not  have  sufficient  personnel  resources  to  handle  the  workload. 
However,  the  contractor's  real  workload  could  be  significantly  reduced 
if  the  recommendations  included  in  this  thesis  are  carried  out  and  a 
higher  percentage  of  ECPs  are  approved  as  written  at  the  CCB. 

Research  the  Time  Requirements.  A  major  assumption  of  this  thesis 
is  that  the  CMP  provided  processing  times  which  could  be  used  as 
standards.  Perhaps  the  times  stated  in  the  CMP  are  unreasonable. 
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Indeed,  DOD-STD-480A  does  not  provide  definite  time  constraints.  A 
study,  conducted  within  Aeronautical  Systems  Division  or  perhaps  within 
Air  Force  Systems  Command,  that  focused  on  ECP  time  requirements,  and 
the  potential  ability  of  organizations  to  meet  them,  could  provide 
useful  results. 

Future  Efforts  Within  the  ACM  SPO.  In  addition  to  performing  time 
assessments,  the  ACM  SPO  could  sponsor  or  conduct  a  future  study  that 
would  include  the  analysis  of  a  survey.  This  survey  would  need  to  be 
prepared  so  that  it  would  not  have  pioblems  with  validity,  as  did  the 
survey  conducted  by  the  CPIT.  The  survey  could  measure  the  SPO's 
feelings  about  how  well  the  ECP  process  is  functioning,  whether  or  not 
effort  is  being  applied  toward  the  TCM,  whether  or  not  the  workload  is 
increasing  or  decreasing,  etc. 

Summary 

In  summary,  this  thesis  has  provided  statistical  evidence  of  an 
untimely  ACM  SPO  ECP  process.  Furthermore,  evidence  was  presented  which 
points  to  the  major  problem  of  a  lack  of  effort  on  the  part  of  people  in 
the  SPO  to  review  the  TCP  and  attend  the  TCM.  This  problem  could 
perhaps  be  reduced  through  the  adoption  of  an  awards  program  which 
recognizes  groups  of  individuals  for  presenting  high-quality  ECPs  at  the 
CCB.  Finally,  the  ACM  SPO  and  others  are  encouraged  to  expend  the 
effort  needed  to  analyze  the  ECP  process  at  other  organizations. 
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Appendix  A.  Roles  of  ACM  SPQ  Functional  Organizations 


The  following  are  brief  descriptions  of  the  roles  the  functional 
organizations  have  in  the  ACM  SPO.  The  following  does  not  include  the 
roles  of  the  Projects,  Dual  Source,  or  Configuration  Management 
Directorates;  their  roles  are  discussed  in  Chapter  II. 

Manufacturing  Management  Directorate  (VCD) . 

Manufacturing  is  responsible  to  the  director  for  managing  all 
manufacturing  activities  involved  in  the  acquisition.  To  develop 
a  product  that  is  in  harmony  with  the  program  objectives,  manufac¬ 
turing  performs  vital  functions  throughout  the  life  cycle  of  the 
weapons  system.  (19:13) 

Engineering  Directorate  (VCE) . 

The  Engineering  Directorate  provides  technical  advice  to  the  PM 
[Program  Manager,  or  Program  Director]  and  manages  the  system 
engineering  function,  including  system  and  subsystem  integration 
and  specialties  engineering  (reliability,  maintainability,  etc.). 
It  provides  system  program  technical  direction  to  the  contractor 
and  assures  the  technical  compatibility  of  all  system  elements- - 
it's  the  Government's  equivalent  to  the  contractor's  engineering 
organization  [and]  provides  engineering  "doers"  as  well  as  the 
contractor  engineering  management  "watchers."  (19:11) 

Safety  Directorate  (VCF) .  VCF  oversees  all  systems  safety 

responsibilities  lor  the  program,  including  any  safety  issues  pertaining 

to  the  current  and  fut  ire  development,  deployment,  operation,  or 

maintenance  of  the  hardware  (16). 

Contracts  Directorate  (VCK) .  The  contracts  directorate  manages 

all  acquisition  activities  of  the  SPO,  especially  those  of  a  contractual 

nature  between  the  government  and  the  contractor  (19:13). 

Intezrated  Logistics  Support  Directorate  (VCL) . 

This  directorate  is  ...  jointly  manned  by  AFLC  [Air  Force  Logis¬ 
tics  Command]  and  AFSC  personnel.  It  provides  the  PM  with  logis¬ 
tical/technical  guidance  and  assistance  in  the  areas  of  relia- 
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bility  and  maintainability;  maintenance  planning;  support  equip¬ 
ment;  supply  support;  packaging,  handling  and  transportation; 
technical  data;  facilities;  manpower  requirements  and  personnel; 
training  and  training  support;  logistic  support  resolve  f-  _,s ; 
logistic  support  management  information;  computer  -  sup¬ 

port;  energy  management;  survivability;  and  ILS  [Integrated  Logis¬ 
tic  Support]  test  and  evaluation.  (19:12) 

Management  Operations  Office  (VCO) . 

This  office  provides  executive  services  and  support  to  the  [SPO] 
in  functions  relating  to  personnel  administration,  publications, 
forms,  and  reports;  training;  correspondence ;  property  and  supply 
control;  and  manpower  and  organization.  (19:13) 

Program  Control  Directorate  fVCP) . 

This  directorate  is  responsible  for  program  planning,  programming, 
progress  tracking,  status  accounting,  trend  analysis,  documenta¬ 
tion,  and  financing.  It  is  the  nerve  center  of  the  SPO  through 
which  the  program  director  maintains  management  control,  surveil¬ 
lance,  and  understanding  of  the  program.  (19:12) 

Security  Directorate  (VCR) .  The  ACM  SPO  is  one  of  the  few  SPOs  in 
ASD  that  has  its  own  professional  security  specialists  on  hand  to 
oversee  all  security  issues  with  respect  to  the  data  and  hardware 
associated  with  the  Advanced  Cruise  Missile  (21).  One  of  VCR's  major 
tasks  is  to  keep  the  Security  Classification  Guide  up  to  date  so 
everyone  who  is  involved  with  the  program  can  stay  current  on  the  clas¬ 
sification  of  aspects  of  the  program  (21) . 

Test  and  Evaluation  Directorate  (VCT) .  The  test  and  evaluation 
function  is  responsible  for  managing  the  overall  testing  effort, 
including  everything  from  developing  system  test  plans  and  objectives  to 
making  sure  test  results  are  analyzed  correctly  and  recommendations  for 
any  corrective  actions  are  made  and  carried  out  (19:12). 

Acquisition  Planning  Directorate  (VCX) .  VCX's  role  is  to  perform 
the  acquisition  planning  function  of  the  program,  including  planning  for 
future  source  selection  activity,  developing  the  acquisition  strategy, 
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i  lit  ep.i  .it  i  ii}’,  the  pi  aim  inj',  activities  for  future  lot  buy;:,  and  piovidinp. 
tlm  potential  view  of  the  future  needed  to  .shape  the  SI'O's  eurient 


ae  t  i  v  i  t  i  es  (  1 '?  )  . 

SKMCO,  Alt  INC, _ and  TASK.  Three  other  organizations.  Systems 

Klip,  i  nee  r  i  up,  and  Manap,enient  Company  (SKMCO),  AKINC  Kesearcli  Corpora 
(AKINC),  and  The  Analytical  Services  Company  (TASK),  provide  teehn 
expertise  to  the  Sl’O,  for  a  price,  throup.h  the  placement  in  the  SI’i 
accord  i  up,  to  their  .specific  specialties  (  1  A )  . 
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Appendix  B.  The  CPU  Survey 


The  following  is  an  example  of  the  survey  administered  to  ACM  SPO 
personnel  by  the  CPIT.  This  questionnaire  was  administered  to  each 
person  in  the  SPO  who  was  involved  with  the  ACM  SPO  ECP  process. 


ACM  CHANGE  PROCESS 
CRITICAL  PROCESS  TEAM  (CPT) 
SURVEY 

ECPs  GOT  YOU  DOWN? 

IS  "EXPEDITE"  A  FOUR-LETTER  WORD? 

DO  YOU  DEFINE  OPR 
"OBSTINATE  PAIN  IN  THE  REAR?" 


Herein  lies  the  beginning  of  improvement.  Much  of  the  framework  for  how 
and  what  the  CPT  attacks  will  be  determined  from  the  results  of  this 
survey.  Please  help  us  to  help  you  by  filling  this  out  honestly  and 
completely.  This  is  a  short  answer  survey  to  maximize  response,  however 
we  welcome  comments,  suggestions,  and  explanations  of  your  answers,  as 
your  time  allows. 

If  you  don't  know  what  the  CPT  is,  you'll  learn  more  about  it  at  the 
January  [1990]  TQM  training.  For  purposes  of  this  survey,  they  are  a 
group  of  ACM  (Govt  &  Contractor)  personnel,  who  are  taking  a  look  at  the 
way  we  process  ECPs/CCPs  with  the  goal  of  making  the  process  easier  for 
everyone  involved.  The  results  of  this  survey  will  be  briefed  at  the 
January  training. 

For  your  information  the  following  people  are  assigned  to  the  CPT: 


Doug  Jones, 

Dave  Parlagreco 
Don  Hoyle 
Patti  Kemper 
Mike  Spencer 
Marilyn  Judd 
Al  Link 


Team  Leader  VCK 

VC  A 

VCL 

VCP 

VCE 

VCC 

VCT 


Teri  Sieck 
Diana  Durdines 
Hal  Lewis 
Teena  Stevens 
Dean  Price 
Jim  Fleming 
Burt  Kurz 


DCASPRO  GD/C 

SEMCO 

ARINC 

SEMCO 

OC-ALC 

MDMSC 

GD/C 


YOUR  NAME  IS  NOT  REQUIRED,  BUT  WE  ASK  THAT  ALL  SPO  MEMBERS  COMPLETE  THE 
SURVEY  AND  TURN  IT  IN  TO  YOUR  POC  BY  COB  18  DEC  89. 
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If  you  don't  understand  a  question  or  have  other  questions  regarding 
this  survey  your  3 -letter  POC  is: 


Please  check  the  item  below  which  most  correctly  describes  your  PRIMARY 
role  in  the  ECP/CCP  process. 

I  am  the  OPR. 


_  I  represent  my  three  letter  organization  on  the  CCB. 

_  I  provide  comments  to  ECPs/CCPs  that  I  am  tasked  to  review. 

_  I  am  responsible  for  coordinating/compiling  the  comments  within 

my  three  letter  organization  for  our  CCB  representative,  but  I 
am  not  the  CCB  representative. 

_  I  am  involved  in  the  administrative  handling  of  ECPs/CCPs  (as 

either  Govt,  or  support  contractor). 

_  I  am  a  support  contractor  providing  review  comments  to  both 

SEMCO  and  the  Government. 

x)  Do  you  feel  that  you  have  been  adequately  trained  to  fulfill  this 
role.  If  not,  how  do  you  feel  you  should  be  trained?  In  what  areas  do 
you  feel  you  need  additional  training? 

x)  If  you  could  change  one  thing  about  the  way  a  ECP/CCP  is  processed 
while  it  is  in  the  SPO,  what  would  it  be? 

x)  If  you  could  change  one  thing  about  the  way  a  ECP/CCP  is  processed 
outside  of  the  SPO,  either  by  the  contractor  or  another  Government 
agency,  what  would  it  be? 

Please  rate  the  following  statements  by  circling  the  number  which  most 
correctly  describes  you  feelings  according  to  the  scale  below: 

1  -  This  is  almost  always  or  always  false 

2  -  This  is  false  most  of  the  time 

3  -  This  is  true  as  often  as  it  is  not  true 

4  -  This  is  true  most  of  the  time 

5  -  This  is  almost  always  or  always  true 

6  -  This  does  not  apply  to  me 

x)  I  have  adequate  time  to  review  ECP/CCPs  and  provide  comments  by  the 
date  requested.  If  you  don't  feel  you  have  adequate  time,  please  write 
in  the  amount  of  time  you  feel  is  necessary. 

1  2  3  4  5  6 

x)  I  attend  the  TCM  for  all  TCPs  that  I  would  comment  on  as  ECP/CCPs. 

1  2  3  4  5  6 
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x)  I  work  to  the  CCB  date  for  an  ECP/CCP  rather  than  the  date  comments 
are  due  to  the  OPR. 

1  2  3  4  5  6 

x)  The  Contractor  has  adequately  addressed  and  documented  the  ECP/CCP 
prior  to  submitting  it  to  the  Government. 

1  2  3  4  5  6 

x)  The  CCB  is  a  formality  with  all  possible  issues  resolved  before  the 
ECP/CCP  is  boarded. 

1  2  3  4  5  6 

x)  My  three  letter  organization  has  a  single  point  of  contract  for  all 
ECP/CCPs . 

1  2  3  4  5  6 

x)  I  am  NOT  experiencing  any  difficulty  with  the  timeliness  of 
companion  ECPs . 

1  2  3  4  5  6 

x)  I  am  NOT  experiencing  any  difficulty  with  the  effectivity  of 
companion  ECPs. 

1  2  3  4  5  6 

x)  The  contractor  complies  with  the  intent  of  the  TCM  minutes. 

1  2  3  4  5  6 

x)  The  contractor  complies  with  the  intent  of  the  CCB  Directive. 

1  2  3  4  5  6 

x)  I  go  to  the  OPR  to  get  my  questions  answered  regarding  an  ECP/CCP 
prior  to  submitting  my  written  input. 

1  2  3  4  5  6 

x)  I  am  informed  of  concessions/changes  made  to  an  ECP/CCP  during 
negotiations  that  impact  me. 

1  2  3  4  5  6 

x)  My  three  letter  organization  disseminates  ECP/CCPs  to  the  proper 
individuals  and  then  forms  a  coordinated  position  for  submittal  to  the 
OPR. 

1  2  3  4  5  6 

x)  The  SPO  has  informal,  but  consistent,  guidelines  for  performing  the 
technical  evaluation  of  ECP/CCPs. 

1  2  3  4  5  6 

x)  I  give  the  OPR  an  opportunity  to  resolve  any  issues  I  have  prior  to 
the  CCB . 

1  2  3  4  5  6 
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x)  The  contractor  implements  TCM  minutes/CCB  directives  as 
agreed/directed. 

1  2  3  4  5  6 

x)  I  know  how  to  find  out  the  status  of  an  ECP/CCP  within  the  SPO. 

1  2  3  4  5  6 


************************************************************************ 


The  following  statements  apply  to  ECP/CCP  OPRs.  If  you  have  never  been 
the  OPR  for  an  ECP/CCP,  skip  to  the  last  section. 

x)  I  am  aware  of  all  issues  prior  to  them  being  raised  at  the  CCB 
regarding  my  ECP/CCPs. 

1  2  3  4  5  6 

x)  I  prefer  that  reviewers  contact  me  to  have  their  issues  resolved 
regarding  my  ECP/CCPs. 

1  2  3  4  5  6 

x)  I  receive  coordinated  comments  by  the  date  requested  from  each  three 
letter  organization. 

1  2  3  4  5  6 

x)  I  know  what  the  established  SPO  procedure  for  margin  submittal  and 
approval  is. 

1  2  3  4  5  6 

x)  I  have  to  go  the  functionals  and  personally  ask  for  their  comments 
in  order  to  compile  then  for  CCB. 

1  2  3  4  5  6 


x)  The  CCB  Representatives  ask  questions  at  the  CCB  that  have  already 
been  resolved  with  their  working  people. 

1  2  3  4  5  6 

x)  I  am  comfortable  with  the  negotiations  process. 


x) 

x) 

X) 

the 

x) 


1 

I  receive 
1 

I  receive 
1 

I  receive 
TCM  where 
1 

I  receive 
1 


2  3  4  5  6 

substantiated  tech  evals  from  the  functionals. 

2  3  4  5  6 

timely  tech  evals  from  the  functionals. 

2  3  4  5  6 

comments  from  functionals  on  ECP/CCPs  who  did  not  attend 
they  could  have  been  resolved. 

2  3  4  5  6 

questions  from  the  functionals  rather  than  statements. 

2  3  4  5  6 
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x)  Some  functionals  consistently  submit  the  same  comments  to  me  on 
every  ECP/CCP. 

1  2  3  4  5  6 

x)  Write  in  Answer:  How  many  days  prior  to  the  CCB  do  you  usually  have 
comments  from  all  functionals?  _  days 

************************************************************************ 

Each  of  the  preceding  statements  addressed  an  issue.  If  you  could  set 
the  priorities  of  the  CPT,  which  issues  (by  statement  number)  would  you 
have  them  address  first: 

Priority  1: _  2: _  3: _ 


************************************************************************ 


Please  use  the  remaining  space  to  annotate  any  additional  concerns , 
comments,  etc.  Attach  additional  sheets  as  required.  Thank  you  for 
your  help. 
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Appendix  C.  Output  from  the  Tests  for  Distribution  Shape 


In  this  appendix  are  the  outputs  from  the  Statistix  and  MathCAD 
computer  programs.  These  outputs  are  examples  of  the  statistical 
examinations  that  were  performed  to  assess  the  distribution  shape  of  the 
data,  whether  normal  or  exponential. 


Test  for  Normal  Distribution 

The  first  test  was  to  determine  if  the  data  followed  the  normal 
distribution.  The  set  of  times  was  entered  into  Statistix  and  the  Wilk- 
Shapiro  test  for  normality  was  invoked  (22:8.4-5).  The  result  was  a 
plot  and  a  calculated  Shapiro-Francia  statistic,  as  shovrn  below  for  the 
set  of  times  between  TCP  Receipt  and  TCM: 


RANKITS  VS  TCPTCM 

RANKITS 
3.0  + 


1.0  + 


-1.0  + 


++ 

2+ 

2  2 

+32+ 

7+ 

2+  22 
43 
+3 


1  + 


+ 


-3.0  + 

-  + . +- 

0.0  20.0 


--  +  — . + . +  - 

40.0  60.0  80.0 

TCPTCM 


APPROX.  WILK- SHAPIRO  0.9478  49  CASES  PLOTTED 
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The  plot  above  shows  an  approximately  straight  line,  and  the  approximate 
Wilk-Shapiro  statistic  is  greater  than  0.90,  so  this  set  of  data  is 
indicated  as  normally  distributed.  However,  if  the  line  was  not 
straight  and  if  the  approximate  Wilk-Shapiro  number  was  not  greater  than 
0.90,  then  the  test  for  the  exponential  distribution  was  employed. 

Test  for  the  Exponential  Distribution 

The  MathCAD  output  is  included  on  the  following  four  pages. 
Basically,  by  entering  a  set  of  process  times,  in  this  case  for  the  time 
between  the  second  CCB  and  receipt  of  the  edited  ECP,  in  the  form  of  a 
vector,  MathCAD  can  be  used  to  produce  not  only  a  histogram  of  the  data 
but  also  the  adjusted  K-S  statistic  value.  The  MathCAD  output  is 
documented,  so  brief  summaries  of  the  steps  of  producing  the  histogram 
and  the  K-S  statistic  follow. 

Histogram .  First,  we  set  up  the  BINS,  which  are  the  boundary 
values  of  the  histogram  intervals  (1:101).  The  minimum  value  of  BINS  is 
the  smallest  value  in  the  "vect"  vector,  and  the  highest  boundary  value 
is  equal  to  the  highest  value  in  "vect"  (1:101).  The  "hist"  function  is 
then  used  to  count  up  the  number  of  data  elements  from  "vect"  that  are 
within  each  of  the  BINS  boundaries  (1:102).  Dividing  the  number  of 
elements  within  each  of  the  boundaries  by  the  total  number  in  "vect" 
produces  a  percentage  which  can  then  be  compared  to  the  theoretically 
determined  percentage  calculated  as  "Ffit."  Furthermore,  by  plotting 
the  theoretical  and  empirical  percentages,  a  preliminary  test  of  the 
distribution  can  be  conducted  (1:103).  Figure  C-l  shows  how  the 
empirical  data  tracks  to  the  theoretical  values.  Next,  we  perform  the 
K-S  test. 
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K-S  Test.  The  MathCAD  output  documents  the  steps  involved  in 


computing  the  adjusted  K-S  test  statistic  value.  The  output  also  shows, 
for  a  range  of  significance  levels,  the  K-S  critical  values.  The 
calculated  values  must  be  less  than  the  critical  values  if  the  data  can 
be  considered  as  following  the  exponential  distribution  (20:195).  A 
figure  also  is  provided  which  shows  the  empirical  and  theoretical 
cumulative  density  functions,  providing  a  visual  comparison  of  the 
goodness  of  fit  between  the  two  cumulative  density  functions. 

Summary 

The  tests  of  the  distribution  shape  were  performed  in  this  manner 
whenever  an  assumption  about  the  shape  of  the  data’s  distribution  shape 
was  made . 
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Test  for  an  exponential  distribution. 


"vect"  is  the  set  of  process  times  for 
the  those  ECPs  that  were  approved  with 
comments  at  the  second  CCB. 


The  sample  statistics  are: 

N  =  7 

mean  =  22.429  the  sample  mean 
pvar  =  348.531  the  population  variance 
svar  =  406.619  the  sample  variance 
ssd  =  20.165  the  sample  standard  deviation 

First,  we  set  up  a  histogram  of  the  data  to  look  at  the 
general  shape  of  the  data.  If  it  appears  to  be  exponential, 
we  press  ahead  with  the  Kolmogorov-Smimov  test.  The 
procedures  for  developing  the  histogram  were  taken  from  the 
MathCAD  User's  Manual  (1:100-108). 


1 

r  -x-x] 

x  :=  1 

..max(vect)  X  :=  - 

mean 

f(x,X)  :=  [X-e  J 

f(x,X)  is  the  equation  for 
the  exponential  distribu¬ 

n  :=  3 

number  of  intervals 

tion,  for  x  >  0. 

I  :=  0 

..n  Index 

MIN  :  = 

min (vect)  first  interval 

MAX  :  = 

max (vect)  max  interval 

j  :=  0  . .n  -  1 

1 

6 

9 

vect  :=  20 

21 
49 
51 


MAX  -  MIN 

BINS  :=  MIN  +  I - 

I  n 

FREQS  :=  hist (BINS, vect) 


p  BINS 

j+1 

Ffit  :=  f(y,X)  dy 

j 

J  BINS 

j 


1 

17.667 
BINS  =  34.333 

51 

FREQS  [0.429' 

-  =  0.286 

N  |_0 . 14  3_ 

0.501 
Ffit  =  0.239 

0.113 
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J:=0..n-1  n  +  1  =  4 


Since  the  data  appears  to  resemble  an  exponential  distribution, 
we  now  perform  the  K-S  test. 

First,  we  must  define  an  empirical  distribution  function,  F(z) , 
where  F(z)  =  (#  Xi's  <=  z)  /  n.  Then,  we  compare  F(z)  to  Ffit 
which  is  the  theoretical  curve  and  is  computed  across  the  domain 
of  vect. 

THE  THEORY 

k  :=  1  . .N 
c  : =  vect 
k  k 


vect 

k 

f(yA)  dy 

0 

Ffit  is  the  continuous  cumulative  density 
function  of  f(x,a,b). 


1 


1 

Ffit 
N  k 


0 

0  d  ,vect  max (vect) 

1  k 


max (vect)  =  51 


Ffit  :  = 
k 


THE  EMPIRICAL 

1  :=  1  .  .N 
d  : =  vect 
1  1 

When  1/n  is  plotted  against  d.l,  then 
the  right-continuous  step  function  is 
generated  (see  plot  below) . 


Figure  C-2.  Plot  of  Theoretical  and  Empirical  Curves 
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k 

E  := - Ffit 

k  N  k 


k  -  1 

F  :=  Ffit  - 

k  k  N 


min(E)  =  -0.03 
max(E)  =  0.106 


min(F)  =  0.036 
max(F)  =  0.173 


maxD  :=  .173  Suprema  Maximum:  the  highest  of  the 
absolute  values  of  the  four  numbers 
above. 


The  calculated  K-S  statistic  for  testing  for  an  exponential 
distribution  is  calculated  from  the  equation  below: 


maxD  -  — 


,26 


.5 

+  — 


■  0.447 


K-S  critical  values  for  exponential  distribution  are: 
equal  to  0.926  0.990  1.094  1.190 
for  (1-Q)  values  of  0.850  0.900  0.950  0.975 


1.308 

0.990 


Since  the  calculated  K-S  statistic  is  LESS  than  the  critical 
value  for  a  significance  level  of  0.90  listed  above,  we  can 
conclude  that  the  data  indicates  an  EXPONENTIAL  fit. 


The  parameters  for  the  data  are: 


X  =  0.045 
mean  =  22.429 
ssd  =  20.165 
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